Sep 10 17:21:48.400865 ip-10-0-129-90 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 17:21:48.400877 ip-10-0-129-90 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 17:21:48.400883 ip-10-0-129-90 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 17:21:48.401157 ip-10-0-129-90 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 17:21:58.285431 ip-10-0-129-90 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 10 17:21:58.285445 ip-10-0-129-90 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 94616a8b3a934388b08e018f48d7e53c --
Sep 10 17:23:48.124818 ip-10-0-129-90 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 17:23:48.523420 ip-10-0-129-90 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 17:23:48.523420 ip-10-0-129-90 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 17:23:48.523420 ip-10-0-129-90 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 17:23:48.523420 ip-10-0-129-90 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 17:23:48.523420 ip-10-0-129-90 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 17:23:48.524378 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.524290    2552 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 17:23:48.528545 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528522    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:23:48.528545 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528542    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:23:48.528545 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528546    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:23:48.528545 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528549    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:23:48.528545 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528552    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528556    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528559    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528562    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528565    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528568    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528571    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528574    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528577    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528579    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528582    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528585    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528588    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528590    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528593    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528595    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528598    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528600    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528603    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528606    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:23:48.528743 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528609    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528611    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528614    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528617    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528619    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528622    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528624    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528630    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528632    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528635    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528638    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528640    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528643    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528645    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528649    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528652    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528655    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528657    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528660    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528662    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:23:48.529223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528665    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528668    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528670    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528674    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528677    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528679    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528682    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528685    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528687    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528690    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528692    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528695    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528698    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528700    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528703    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528705    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528708    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528710    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528713    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528715    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:23:48.529726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528718    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528720    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528723    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528725    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528728    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528730    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528734    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528737    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528739    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528742    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528745    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528747    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528752    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528756    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528759    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528762    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528765    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528776    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528781    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:23:48.530229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528785    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528789    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528793    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528796    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528799    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528802    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528805    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528808    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528811    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528814    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528817    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528819    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528822    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528825    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528828    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.528831    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529529    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529537    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529540    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:23:48.530745 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529543    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529547    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529550    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529553    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529556    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529559    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529562    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529565    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529568    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529571    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529574    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529576    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529579    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529582    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529586    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529590    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529593    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529596    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529599    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:23:48.531213 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529601    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529604    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529607    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529610    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529613    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529617    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529620    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529623    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529626    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529629    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529632    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529634    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529637    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529641    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529644    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529647    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529649    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529652    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529655    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:23:48.531693 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529658    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529660    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529663    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529665    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529668    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529670    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529673    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529675    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529677    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529681    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529684    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529687    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529689    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529692    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529694    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529697    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529699    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529702    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529705    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529707    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:23:48.532162 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529710    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529712    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529715    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529718    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529720    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529722    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529729    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529732    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529734    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529737    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529740    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529744    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529747    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529750    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529753    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529756    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529759    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529761    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529764    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529766    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:23:48.532666 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529769    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529771    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529774    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529778    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529781    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529783    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529786    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529788    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529791    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529793    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529796    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529799    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529801    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529804    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529806    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529809    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529811    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.529814    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530200    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530205    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:23:48.533153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530208    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530211    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530214    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530216    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530220    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530223    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530225    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530228    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530231    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530234    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530237    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530239    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530242    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530245    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530248    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530250    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530253    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530256    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530258    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530261    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:23:48.533736 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530263    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530266    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530268    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530271    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530274    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530277    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530279    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530282    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530285    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530288    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530292    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530295    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530298    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530300    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530303    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530305    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530308    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530310    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530313    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:23:48.534227 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530315    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530318    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530320    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530325    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530327    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530330    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530332    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530335    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530338    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530340    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530343    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530346    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530348    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530351    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530353    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530356    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530359    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530361    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530363    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530366    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:23:48.534710 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530369    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530372    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530374    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530378    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530381    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530383    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530386    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530389    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530391    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530394    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530396    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530399    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530401    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530404    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530406    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530410    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530412    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530416    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530420    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530423    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:23:48.535187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530426    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530430    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530434    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530437    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530441    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530444    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530446    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530449    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530452    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530455    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530458    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530460    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530463    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530466    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530468    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530472    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530476    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.530479    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530571    2552 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530610    2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 17:23:48.535713 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530617    2552 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530622    2552 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530633    2552 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530638    2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530644    2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530650    2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530655    2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530665    2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530670    2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530674    2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530677    2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530680    2552 flags.go:64] FLAG: --cgroup-root=""
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530683    2552 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530686    2552 flags.go:64] FLAG: --client-ca-file=""
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530690    2552 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530693    2552 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530697    2552 flags.go:64] FLAG: --cluster-domain=""
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530700    2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530704    2552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530706    2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530710    2552 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530713    2552 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530716    2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 17:23:48.536200 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530719    2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530722    2552 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530725    2552 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530728    2552 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530732    2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530735    2552 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530738    2552 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530742    2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530745    2552 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530748    2552 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530751    2552 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530754    2552 flags.go:64] FLAG: --enable-server="true"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530757    2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530761    2552 flags.go:64] FLAG: --event-burst="100"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530765    2552 flags.go:64] FLAG: --event-qps="50"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530767    2552 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530772    2552 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530775    2552 flags.go:64] FLAG: --eviction-hard=""
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530778    2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530781    2552 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530784    2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530787    2552 flags.go:64] FLAG: --eviction-soft=""
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530790    2552 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530793    2552 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530796    2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 17:23:48.536763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530799    2552 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530802    2552 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530805    2552 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530807    2552 flags.go:64] FLAG: --feature-gates=""
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530811    2552 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530814    2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530817    2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530820    2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530823    2552 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530826    2552 flags.go:64] FLAG: --help="false"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530828    2552 flags.go:64] FLAG: --hostname-override="ip-10-0-129-90.ec2.internal"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530832    2552 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530835    2552 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530838    2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530841    2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530844    2552 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530847    2552 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530850    2552 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530853    2552 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530856    2552 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530858    2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530861    2552 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530864    2552 flags.go:64] FLAG: --kube-reserved=""
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530867    2552 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 17:23:48.537399 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530870    2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530873    2552 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530876    2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530879    2552 flags.go:64] FLAG: --lock-file=""
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530881    2552 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530884    2552 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530887    2552 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530892    2552 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530895    2552 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530898    2552 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530901    2552 flags.go:64] FLAG: --logging-format="text"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530904    2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530907    2552 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530910    2552 flags.go:64] FLAG: --manifest-url=""
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530913    2552 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530917    2552 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530920    2552 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530924    2552 flags.go:64] FLAG: --max-pods="110"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530927    2552 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530930    2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530933    2552 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530936    2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530939    2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530942    2552 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530945    2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 17:23:48.537992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530952    2552 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530955    2552 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530958    2552 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530961    2552 flags.go:64] FLAG: --pod-cidr=""
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530964    2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530969    2552 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530972    2552 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530975    2552 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530978    2552 flags.go:64] FLAG: --port="10250"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530981    2552 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530984    2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06a110f3c7eeb943e"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530987    2552 flags.go:64] FLAG: --qos-reserved=""
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530990    2552 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530993    2552 flags.go:64] FLAG: --register-node="true"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530996    2552 flags.go:64] FLAG: --register-with-taints=""
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.530999    2552 flags.go:64] FLAG: --registry-burst="10"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531003    2552 flags.go:64] FLAG: --registry-qps="5"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531006    2552 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531008    2552 flags.go:64] FLAG: --reserved-memory=""
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531012    2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531015    2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531018    2552 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531021    2552 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531024    2552 flags.go:64] FLAG: --runonce="false"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531027    2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 17:23:48.538614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531030    2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531033    2552 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531036    2552 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531039    2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531042    2552 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531045    2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531048    2552 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531051    2552 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531054    2552 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531057    2552 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531060    2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531063    2552 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531065    2552 flags.go:64] FLAG: --system-cgroups=""
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531068    2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531074    2552 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531076    2552 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531079    2552 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531084    2552 flags.go:64] FLAG: --tls-min-version=""
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531087    2552 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531089    2552 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531092    2552 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531095    2552 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531098    2552 flags.go:64] FLAG: --v="2"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531102    2552 flags.go:64] FLAG: --version="false"
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531106    2552 flags.go:64] FLAG: --vmodule=""
Sep 10 17:23:48.539269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531110    2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.531113    2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531226    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531230    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531233    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531236    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531238    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531241    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531243    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531246    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531248    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531252    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531255    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531258    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531260    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531263    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531265    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531268    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531271    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531273    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:23:48.539872 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531277    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531279    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531282    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531284    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531287    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531289    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531292    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531294    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531297    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531299    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531302    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531305    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531308    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531311    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531313    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531316    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531318    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531321    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531323    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531326    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531329    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:23:48.540356 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531332    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531335    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531337    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531340    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531343    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531345    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531348    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531350    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531353    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531355    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531358    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531362    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531364    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531367    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531370    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531373    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531375    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531378    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531382    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:23:48.540902 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531386    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531389    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531392    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531395    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531397    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531400    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531403    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531405    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531408    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531410    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531413    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531415    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531418    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531422    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531425    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531427    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531430    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531432    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531435    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:23:48.541365 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531438    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531440    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531443    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531445    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531448    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531453    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531456    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531459    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531462    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531465    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531468    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531470    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531483    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531486    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531489    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531505    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531510    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531514    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531517    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:23:48.541837 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531520    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:23:48.542304 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531523    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:23:48.542304 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.531526    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:23:48.542304 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.532124    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 17:23:48.542304 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.539162    2552 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 17:23:48.542304 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.539177    2552 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 17:23:48.542304 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.540001    2552 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 17:23:48.542304 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.540008    2552 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 17:23:48.542304 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540065    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:23:48.542304 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540071    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:23:48.542304 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540074    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:23:48.542304 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540078    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:23:48.542304 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540082    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:23:48.542304 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540086    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:23:48.542304 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540089    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540091    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540094    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540097    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540099    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540102    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540105    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540108    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540110    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540113    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540115    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540118    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540121    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540123    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540126    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540129    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540131    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540134    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540137    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540140    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:23:48.542784 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540143    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540145    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540148    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540151    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540154    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540157    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540160    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540162    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540165    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540167    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540170    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540172    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540175    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540178    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540181    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540184    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540187    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540190    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540192    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540195    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:23:48.543261 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540197    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540200    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540202    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540205    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540208    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540210    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540213    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540215    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540218    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540221    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540223    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540226    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540229    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540231    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540234    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540238    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540240    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540243    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540245    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540248    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:23:48.543755 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540250    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540253    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540255    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540258    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540260    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540263    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540265    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540268    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540270    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540274    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540278    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540281    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540284    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540287    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540289    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540292    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540294    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540297    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540299    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540302    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:23:48.544229 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540305    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:23:48.544723 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540307    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:23:48.544723 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540310    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:23:48.544723 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540312    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:23:48.544723 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540316    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:23:48.544723 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540320    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:23:48.544723 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540323    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:23:48.544723 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540326    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:23:48.544723 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540329    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:23:48.544723 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540331    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:23:48.544723 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540333    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:23:48.544723 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540336    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:23:48.544723 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540339    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:23:48.544723 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.540344    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 17:23:48.544723 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540451    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:23:48.544723 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540456    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540459    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540462    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540465    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540467    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540470    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540474    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540478    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540481    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540484    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540487    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540500    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540503    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540506    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540509    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540511    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540514    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540517    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540520    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:23:48.545091 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540523    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540526    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540528    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540531    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540534    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540537    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540540    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540543    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540546    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540548    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540551    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540553    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540556    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540558    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540561    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540563    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540566    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540569    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540571    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540574    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:23:48.545555 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540576    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540579    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540582    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540584    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540587    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540589    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540592    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540595    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540597    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540600    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540602    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540605    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540607    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540609    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540612    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540614    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540617    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540620    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540623    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540626    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:23:48.546032 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540629    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540631    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540633    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540636    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540638    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540641    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540644    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540646    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540649    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540652    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540656    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540659    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540662    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540665    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540668    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540671    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540673    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540676    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540678    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:23:48.546588 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540681    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540683    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540685    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540688    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540691    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540693    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540696    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540698    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540701    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540703    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540706    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540709    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540712    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540714    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540717    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540719    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540722    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540724    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540727    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:23:48.547043 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:48.540729    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:23:48.547489 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.540734    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 17:23:48.547489 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.540841    2552 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 17:23:48.547489 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.542739    2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 17:23:48.547489 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.543800    2552 server.go:1019] "Starting client certificate rotation"
Sep 10 17:23:48.547489 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.543894    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 17:23:48.547489 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.544794    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 17:23:48.547489 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.546177    2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 17:23:48.548114 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.548096    2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 17:23:48.558951 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.558928    2552 log.go:25] "Validated CRI v1 runtime API"
Sep 10 17:23:48.561837 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.561818    2552 log.go:25] "Validated CRI v1 image API"
Sep 10 17:23:48.562701 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.562673    2552 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 17:23:48.565314 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.565293    2552 fs.go:135] Filesystem UUIDs: map[2d2103f8-77a5-4ed3-b222-d11e954d27b3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b543cc24-233b-4481-a99c-89a4c9f75503:/dev/nvme0n1p3]
Sep 10 17:23:48.565376 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.565314    2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 17:23:48.570623 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.570503    2552 manager.go:216] Machine: {Timestamp:2026-09-10 17:23:48.569224553 +0000 UTC m=+0.327751617 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099068 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a9da9e0d0e645d0629228455e0d5b SystemUUID:ec2a9da9-e0d0-e645-d062-9228455e0d5b BootID:94616a8b-3a93-4388-b08e-018f48d7e53c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1f:6c:d2:2c:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1f:6c:d2:2c:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:05:bd:4d:d0:b9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 17:23:48.570623 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.570622    2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 17:23:48.570773 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.570706    2552 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 17:23:48.572293 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.572258    2552 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 17:23:48.572431 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.572297    2552 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-90.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 17:23:48.573027 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.573016    2552 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 17:23:48.573061 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.573030    2552 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 17:23:48.573061 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.573038    2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 17:23:48.573740 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.573729    2552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 17:23:48.574900 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.574885    2552 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 17:23:48.577331 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.577314    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 17:23:48.577981 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.577970    2552 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 17:23:48.578104 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.578095    2552 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 17:23:48.578137 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.578126    2552 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 17:23:48.578168 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.578141    2552 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 17:23:48.578168 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.578152    2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 17:23:48.578168 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.578161    2552 kubelet.go:398] "Adding apiserver pod source"
Sep 10 17:23:48.578251 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.578171    2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 17:23:48.578538 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.578526    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 17:23:48.578572 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.578551    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 17:23:48.580987 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.580971    2552 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 17:23:48.583156 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.583142    2552 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 17:23:48.583204 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.583173    2552 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 17:23:48.583320 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.583310    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 17:23:48.583362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.583325    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 17:23:48.583362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.583332    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 17:23:48.583362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.583337    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 17:23:48.583362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.583346    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 17:23:48.583362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.583354    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 17:23:48.583898 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.583888    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 17:23:48.583898 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.583898    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 17:23:48.583966 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.583904    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 17:23:48.583966 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.583921    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 17:23:48.583966 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.583937    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 17:23:48.583966 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.583944    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 17:23:48.584481 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.584472    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 17:23:48.584530 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.584482    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 17:23:48.588151 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.588136    2552 server.go:1268] "Started kubelet"
Sep 10 17:23:48.588305 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.588254    2552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 17:23:48.588362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.588303    2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 17:23:48.588362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.588332    2552 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 17:23:48.589342 ip-10-0-129-90 systemd[1]: Started Kubernetes Kubelet.
Sep 10 17:23:48.589583 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.589488    2552 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 17:23:48.589942 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.589894    2552 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-90.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 17:23:48.590030 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.589945    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 17:23:48.590030 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.589972    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-90.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 17:23:48.592752 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.592719    2552 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 17:23:48.596319 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.596290    2552 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 17:23:48.596458 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.596446    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 17:23:48.596998 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.596971    2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 17:23:48.597208 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.597168    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-90.ec2.internal\" not found"
Sep 10 17:23:48.597777 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.597759    2552 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 17:23:48.597777 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.597765    2552 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 17:23:48.597891 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.597785    2552 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 17:23:48.597891 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.597805    2552 factory.go:55] Registering systemd factory
Sep 10 17:23:48.597891 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.597821    2552 factory.go:223] Registration of the systemd container factory successfully
Sep 10 17:23:48.597891 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.597854    2552 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 17:23:48.597891 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.597859    2552 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 17:23:48.598114 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.598000    2552 factory.go:153] Registering CRI-O factory
Sep 10 17:23:48.598114 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.598009    2552 factory.go:223] Registration of the crio container factory successfully
Sep 10 17:23:48.598114 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.598053    2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 17:23:48.598114 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.598066    2552 factory.go:103] Registering Raw factory
Sep 10 17:23:48.598114 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.598077    2552 manager.go:1196] Started watching for new ooms in manager
Sep 10 17:23:48.598377 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.598366    2552 manager.go:349] Starting recovery of all containers
Sep 10 17:23:48.599539 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.598466    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-90.ec2.internal.18d40586d27d4258  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-90.ec2.internal,UID:ip-10-0-129-90.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-90.ec2.internal,},FirstTimestamp:2026-09-10 17:23:48.588110424 +0000 UTC m=+0.346637486,LastTimestamp:2026-09-10 17:23:48.588110424 +0000 UTC m=+0.346637486,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-90.ec2.internal,}"
Sep 10 17:23:48.602528 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.602483    2552 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-90.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 17:23:48.602602 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.602578    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 17:23:48.607541 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.607525    2552 manager.go:354] Recovery completed
Sep 10 17:23:48.613707 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.613694    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:23:48.615537 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.615520    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:23:48.615608 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.615549    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:23:48.615608 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.615560    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:23:48.616133 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.616120    2552 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 17:23:48.616133 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.616131    2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 17:23:48.616242 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.616135    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:23:48.616242 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.616145    2552 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 17:23:48.616910 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.616895    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:23:48.616966 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.616926    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:23:48.616966 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.616939    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:23:48.617268 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.617256    2552 policy_none.go:49] "None policy: Start"
Sep 10 17:23:48.617301 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.617272    2552 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 17:23:48.617301 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.617282    2552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 17:23:48.617752 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.617669    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-90.ec2.internal.18d40586d41fbdf3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-90.ec2.internal,UID:ip-10-0-129-90.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-90.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-90.ec2.internal,},FirstTimestamp:2026-09-10 17:23:48.615536115 +0000 UTC m=+0.374063176,LastTimestamp:2026-09-10 17:23:48.615536115 +0000 UTC m=+0.374063176,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-90.ec2.internal,}"
Sep 10 17:23:48.618248 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.618237    2552 policy_none.go:47] "Start"
Sep 10 17:23:48.627389 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.627214    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-90.ec2.internal.18d40586d420040c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-90.ec2.internal,UID:ip-10-0-129-90.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-90.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-90.ec2.internal,},FirstTimestamp:2026-09-10 17:23:48.61555406 +0000 UTC m=+0.374081121,LastTimestamp:2026-09-10 17:23:48.61555406 +0000 UTC m=+0.374081121,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-90.ec2.internal,}"
Sep 10 17:23:48.632593 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.631137    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jfpw8"
Sep 10 17:23:48.639888 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.639864    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jfpw8"
Sep 10 17:23:48.641580 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.641487    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-90.ec2.internal.18d40586d4202c7d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-90.ec2.internal,UID:ip-10-0-129-90.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-90.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-90.ec2.internal,},FirstTimestamp:2026-09-10 17:23:48.615564413 +0000 UTC m=+0.374091474,LastTimestamp:2026-09-10 17:23:48.615564413 +0000 UTC m=+0.374091474,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-90.ec2.internal,}"
Sep 10 17:23:48.643115 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.643101    2552 manager.go:337] "Starting Device Plugin manager"
Sep 10 17:23:48.643179 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.643141    2552 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 17:23:48.643179 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.643154    2552 server.go:87] "Starting device plugin registration server"
Sep 10 17:23:48.643450 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.643438    2552 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 17:23:48.643531 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.643450    2552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 17:23:48.643650 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.643625    2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 17:23:48.643740 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.643715    2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 17:23:48.643740 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.643724    2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 17:23:48.644218 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.644199    2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 17:23:48.644287 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.644249    2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-90.ec2.internal\" not found"
Sep 10 17:23:48.649281 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.649199    2552 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-90.ec2.internal.18d40586d41fbdf3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-90.ec2.internal.18d40586d41fbdf3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-90.ec2.internal,UID:ip-10-0-129-90.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-90.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-90.ec2.internal,},FirstTimestamp:2026-09-10 17:23:48.615536115 +0000 UTC m=+0.374063176,LastTimestamp:2026-09-10 17:23:48.616910287 +0000 UTC m=+0.375437348,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-90.ec2.internal,}"
Sep 10 17:23:48.660434 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.660342    2552 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-90.ec2.internal.18d40586d420040c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-90.ec2.internal.18d40586d420040c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-90.ec2.internal,UID:ip-10-0-129-90.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-90.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-90.ec2.internal,},FirstTimestamp:2026-09-10 17:23:48.61555406 +0000 UTC m=+0.374081121,LastTimestamp:2026-09-10 17:23:48.616932082 +0000 UTC m=+0.375459147,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-90.ec2.internal,}"
Sep 10 17:23:48.763602 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.671739    2552 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-90.ec2.internal.18d40586d4202c7d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-90.ec2.internal.18d40586d4202c7d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-90.ec2.internal,UID:ip-10-0-129-90.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-90.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-90.ec2.internal,},FirstTimestamp:2026-09-10 17:23:48.615564413 +0000 UTC m=+0.374091474,LastTimestamp:2026-09-10 17:23:48.616944628 +0000 UTC m=+0.375471690,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-90.ec2.internal,}"
Sep 10 17:23:48.763602 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.680142    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-90.ec2.internal.18d40586d5f7e2af  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-90.ec2.internal,UID:ip-10-0-129-90.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-129-90.ec2.internal,},FirstTimestamp:2026-09-10 17:23:48.646478511 +0000 UTC m=+0.405005564,LastTimestamp:2026-09-10 17:23:48.646478511 +0000 UTC m=+0.405005564,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-90.ec2.internal,}"
Sep 10 17:23:48.763602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.744440    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:23:48.763602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.745694    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:23:48.763602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.745732    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:23:48.763602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.745746    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:23:48.763602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.745773    2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-90.ec2.internal"
Sep 10 17:23:48.763602 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.755456    2552 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-90.ec2.internal.18d40586d41fbdf3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-90.ec2.internal.18d40586d41fbdf3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-90.ec2.internal,UID:ip-10-0-129-90.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-90.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-90.ec2.internal,},FirstTimestamp:2026-09-10 17:23:48.615536115 +0000 UTC m=+0.374063176,LastTimestamp:2026-09-10 17:23:48.745715156 +0000 UTC m=+0.504242216,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-90.ec2.internal,}"
Sep 10 17:23:48.764605 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.764537    2552 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-90.ec2.internal.18d40586d420040c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-90.ec2.internal.18d40586d420040c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-90.ec2.internal,UID:ip-10-0-129-90.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-90.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-90.ec2.internal,},FirstTimestamp:2026-09-10 17:23:48.61555406 +0000 UTC m=+0.374081121,LastTimestamp:2026-09-10 17:23:48.745738659 +0000 UTC m=+0.504265725,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-90.ec2.internal,}"
Sep 10 17:23:48.764731 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.764711    2552 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-90.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-129-90.ec2.internal"
Sep 10 17:23:48.795471 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.795384    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 17:23:48.796586 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.796569    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 17:23:48.796651 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.796619    2552 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 17:23:48.796651 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.796640    2552 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 17:23:48.796737 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.796724    2552 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 17:23:48.798649 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.798630    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:23:48.808305 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.808285    2552 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-90.ec2.internal\" not found" node="ip-10-0-129-90.ec2.internal"
Sep 10 17:23:48.898259 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.897461    2552 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-90.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal"]
Sep 10 17:23:48.898352 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.898302    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:23:48.898752 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.898734    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9967cb0cd42cd2e177721f6d62bd6a5e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-90.ec2.internal\" (UID: \"9967cb0cd42cd2e177721f6d62bd6a5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-90.ec2.internal"
Sep 10 17:23:48.899335 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.899317    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:23:48.899428 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.899347    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:23:48.899428 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.899357    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:23:48.900592 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.900579    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:23:48.900718 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.900700    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-90.ec2.internal"
Sep 10 17:23:48.900788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.900739    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:23:48.901320 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.901305    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:23:48.901403 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.901334    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:23:48.901403 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.901348    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:23:48.901916 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.901892    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:23:48.901916 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.901918    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:23:48.902036 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.901930    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:23:48.903169 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.903157    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal"
Sep 10 17:23:48.903217 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.903182    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:23:48.903849 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.903836    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:23:48.903914 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.903862    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:23:48.903914 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.903877    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:23:48.932320 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.932297    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-90.ec2.internal\" not found" node="ip-10-0-129-90.ec2.internal"
Sep 10 17:23:48.936708 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.936692    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-90.ec2.internal\" not found" node="ip-10-0-129-90.ec2.internal"
Sep 10 17:23:48.965542 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.965522    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:23:48.966402 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.966387    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:23:48.966481 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.966418    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:23:48.966481 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.966432    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:23:48.966481 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.966463    2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-90.ec2.internal"
Sep 10 17:23:48.974669 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.974653    2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-90.ec2.internal"
Sep 10 17:23:48.974744 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.974675    2552 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-90.ec2.internal\": node \"ip-10-0-129-90.ec2.internal\" not found"
Sep 10 17:23:48.999051 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.999024    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9967cb0cd42cd2e177721f6d62bd6a5e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-90.ec2.internal\" (UID: \"9967cb0cd42cd2e177721f6d62bd6a5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-90.ec2.internal"
Sep 10 17:23:48.999182 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.999059    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37cc8c335d286fe4e7fe52fe1d7f1f34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal\" (UID: \"37cc8c335d286fe4e7fe52fe1d7f1f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal"
Sep 10 17:23:48.999182 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.999077    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37cc8c335d286fe4e7fe52fe1d7f1f34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal\" (UID: \"37cc8c335d286fe4e7fe52fe1d7f1f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal"
Sep 10 17:23:48.999182 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:48.999128    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9967cb0cd42cd2e177721f6d62bd6a5e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-90.ec2.internal\" (UID: \"9967cb0cd42cd2e177721f6d62bd6a5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-90.ec2.internal"
Sep 10 17:23:48.999710 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:48.999696    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-90.ec2.internal\" not found"
Sep 10 17:23:49.099237 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.099189    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37cc8c335d286fe4e7fe52fe1d7f1f34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal\" (UID: \"37cc8c335d286fe4e7fe52fe1d7f1f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal"
Sep 10 17:23:49.099237 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.099147    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37cc8c335d286fe4e7fe52fe1d7f1f34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal\" (UID: \"37cc8c335d286fe4e7fe52fe1d7f1f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal"
Sep 10 17:23:49.099394 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.099267    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37cc8c335d286fe4e7fe52fe1d7f1f34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal\" (UID: \"37cc8c335d286fe4e7fe52fe1d7f1f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal"
Sep 10 17:23:49.099394 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.099315    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37cc8c335d286fe4e7fe52fe1d7f1f34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal\" (UID: \"37cc8c335d286fe4e7fe52fe1d7f1f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal"
Sep 10 17:23:49.100206 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:49.100188    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-90.ec2.internal\" not found"
Sep 10 17:23:49.200712 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:49.200684    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-90.ec2.internal\" not found"
Sep 10 17:23:49.234893 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.234859    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-90.ec2.internal"
Sep 10 17:23:49.239085 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.238956    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal"
Sep 10 17:23:49.300793 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:49.300754    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-90.ec2.internal\" not found"
Sep 10 17:23:49.400893 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:49.400827    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-90.ec2.internal\" not found"
Sep 10 17:23:49.403279 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.403255    2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:23:49.493371 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.493336    2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:23:49.501813 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:49.501787    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-90.ec2.internal\" not found"
Sep 10 17:23:49.543118 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.543086    2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 17:23:49.543648 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.543218    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 17:23:49.543648 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.543252    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 17:23:49.543648 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.543256    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 17:23:49.602301 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:49.602269    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-90.ec2.internal\" not found"
Sep 10 17:23:49.703078 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:49.702991    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-90.ec2.internal\" not found"
Sep 10 17:23:49.720395 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.720369    2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:23:49.763537 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.763486    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 17:18:48 +0000 UTC" deadline="2028-04-06 10:21:04.732096841 +0000 UTC"
Sep 10 17:23:49.763643 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.763541    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13768h57m14.968561473s"
Sep 10 17:23:49.798244 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.798209    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-90.ec2.internal"
Sep 10 17:23:49.807540 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.807505    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 17:23:49.808994 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.808979    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal"
Sep 10 17:23:49.820975 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.820953    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 17:23:49.909120 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:49.909085    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cc8c335d286fe4e7fe52fe1d7f1f34.slice/crio-7bd45c2bda6dd6544f8de2ab911564544f83651dd9d615c08ee1e730772dbad4 WatchSource:0}: Error finding container 7bd45c2bda6dd6544f8de2ab911564544f83651dd9d615c08ee1e730772dbad4: Status 404 returned error can't find the container with id 7bd45c2bda6dd6544f8de2ab911564544f83651dd9d615c08ee1e730772dbad4
Sep 10 17:23:49.909342 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:49.909320    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9967cb0cd42cd2e177721f6d62bd6a5e.slice/crio-538e73f39d00912c204bf4469dddd1a34704b23a55f36e82c718aa47eebc68b9 WatchSource:0}: Error finding container 538e73f39d00912c204bf4469dddd1a34704b23a55f36e82c718aa47eebc68b9: Status 404 returned error can't find the container with id 538e73f39d00912c204bf4469dddd1a34704b23a55f36e82c718aa47eebc68b9
Sep 10 17:23:49.914649 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:49.913874    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:23:50.579398 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.579364    2552 apiserver.go:52] "Watching apiserver"
Sep 10 17:23:50.587734 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.587704    2552 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 17:23:50.589975 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.589952    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-bbbdj","kube-system/konnectivity-agent-ndkbc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68","openshift-image-registry/node-ca-wwbzr","openshift-multus/multus-additional-cni-plugins-ndldz","openshift-multus/network-metrics-daemon-jk4wd","openshift-network-diagnostics/network-check-target-9hknl","kube-system/kube-apiserver-proxy-ip-10-0-129-90.ec2.internal","openshift-cluster-node-tuning-operator/tuned-b87fg","openshift-dns/node-resolver-f9tg8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal","openshift-multus/multus-8h6xk","openshift-network-operator/iptables-alerter-vnd6t","openshift-ovn-kubernetes/ovnkube-node-vcnth"]
Sep 10 17:23:50.591637 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.591614    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:50.591746 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:50.591692    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:23:50.592737 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.592713    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ndkbc"
Sep 10 17:23:50.594170 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.594005    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.595256 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.595232    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wwbzr"
Sep 10 17:23:50.595744 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.595723    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ttrt7\""
Sep 10 17:23:50.596317 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.596290    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 17:23:50.596577 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.596559    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 17:23:50.596685 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.596650    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 17:23:50.596843 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.596802    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-brdfz\""
Sep 10 17:23:50.596915 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.596863    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 17:23:50.597317 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.597299    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 17:23:50.597569 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.597545    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 17:23:50.597766 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.597574    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 17:23:50.597766 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.597550    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.597766 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.597760    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 17:23:50.598417 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.598267    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sjxpm\""
Sep 10 17:23:50.598751 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.598724    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 17:23:50.599748 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.599541    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:23:50.599748 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:50.599611    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:23:50.600898 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.600861    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:23:50.601004 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:50.600933    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:23:50.602868 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.602561    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.603071 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.603049    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 17:23:50.603263 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.603249    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 17:23:50.603430 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.603406    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 17:23:50.603541 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.603435    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ktq8c\""
Sep 10 17:23:50.603610 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.603567    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 17:23:50.603908 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.603889    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f9tg8"
Sep 10 17:23:50.604319 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.604291    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 17:23:50.605115 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605011    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btlsg\" (UniqueName: \"kubernetes.io/projected/7522eaf2-4eaa-461a-9681-3df368665161-kube-api-access-btlsg\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.605115 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605049    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7067d92a-5363-42fe-a8f0-7c765931c532-host\") pod \"node-ca-wwbzr\" (UID: \"7067d92a-5363-42fe-a8f0-7c765931c532\") " pod="openshift-image-registry/node-ca-wwbzr"
Sep 10 17:23:50.605115 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605076    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lczp8\" (UniqueName: \"kubernetes.io/projected/7067d92a-5363-42fe-a8f0-7c765931c532-kube-api-access-lczp8\") pod \"node-ca-wwbzr\" (UID: \"7067d92a-5363-42fe-a8f0-7c765931c532\") " pod="openshift-image-registry/node-ca-wwbzr"
Sep 10 17:23:50.605115 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605111    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d6869e8-701f-4a3f-85c6-7ee650066984-cnibin\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.605360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605137    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d6869e8-701f-4a3f-85c6-7ee650066984-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.605360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605161    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4dkx\" (UniqueName: \"kubernetes.io/projected/4d6869e8-701f-4a3f-85c6-7ee650066984-kube-api-access-n4dkx\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.605360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605217    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed270dc5-1f93-4739-b3b1-d5a1412d6819-kubelet-config\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:50.605360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605241    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:50.605360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605265    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-socket-dir\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.605360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605291    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d6869e8-701f-4a3f-85c6-7ee650066984-cni-binary-copy\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.605360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605330    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d6869e8-701f-4a3f-85c6-7ee650066984-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.605703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605367    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqn8x\" (UniqueName: \"kubernetes.io/projected/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-kube-api-access-cqn8x\") pod \"network-metrics-daemon-jk4wd\" (UID: \"3e821fe8-fb34-458c-b7d0-521a7ad7a31e\") " pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:23:50.605703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605392    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.605703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605417    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-device-dir\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.605703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605454    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:23:50.605703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605394    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 17:23:50.605703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605664    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7r266\""
Sep 10 17:23:50.605982 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605453    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-etc-selinux\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.605982 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605810    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-sys-fs\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.605982 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605838    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d6869e8-701f-4a3f-85c6-7ee650066984-system-cni-dir\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.605982 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605865    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d6869e8-701f-4a3f-85c6-7ee650066984-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.605982 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605888    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17564146-f078-4b05-a08f-2ac7d4f1d60d-konnectivity-ca\") pod \"konnectivity-agent-ndkbc\" (UID: \"17564146-f078-4b05-a08f-2ac7d4f1d60d\") " pod="kube-system/konnectivity-agent-ndkbc"
Sep 10 17:23:50.605982 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605910    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-registration-dir\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.605982 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605934    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7067d92a-5363-42fe-a8f0-7c765931c532-serviceca\") pod \"node-ca-wwbzr\" (UID: \"7067d92a-5363-42fe-a8f0-7c765931c532\") " pod="openshift-image-registry/node-ca-wwbzr"
Sep 10 17:23:50.605982 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605965    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs\") pod \"network-metrics-daemon-jk4wd\" (UID: \"3e821fe8-fb34-458c-b7d0-521a7ad7a31e\") " pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:23:50.606393 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.605987    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed270dc5-1f93-4739-b3b1-d5a1412d6819-dbus\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:50.606393 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.606020    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17564146-f078-4b05-a08f-2ac7d4f1d60d-agent-certs\") pod \"konnectivity-agent-ndkbc\" (UID: \"17564146-f078-4b05-a08f-2ac7d4f1d60d\") " pod="kube-system/konnectivity-agent-ndkbc"
Sep 10 17:23:50.606393 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.606148    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.607601 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.607302    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vnd6t"
Sep 10 17:23:50.610473 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.610373    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.611871 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.611853    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 17:23:50.611871 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.611863    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 17:23:50.612197 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.612177    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 17:23:50.613170 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.613150    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 17:23:50.613340 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.613320    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-spzjl\""
Sep 10 17:23:50.613871 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.613852    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 17:23:50.614439 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.614422    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qgqgm\""
Sep 10 17:23:50.615108 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.614606    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:23:50.615108 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.614820    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 17:23:50.615358 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.615211    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 17:23:50.615358 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.615257    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-749sc\""
Sep 10 17:23:50.616969 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.615448    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 17:23:50.616969 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.615656    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 17:23:50.616969 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.615704    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 17:23:50.616969 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.616182    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 17:23:50.617465 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.617075    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nqlkp\""
Sep 10 17:23:50.618753 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.618727    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 17:23:50.646122 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.646099    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4qhnb"
Sep 10 17:23:50.656749 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.656726    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4qhnb"
Sep 10 17:23:50.699527 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.699444    2552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 17:23:50.707239 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707210    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17564146-f078-4b05-a08f-2ac7d4f1d60d-agent-certs\") pod \"konnectivity-agent-ndkbc\" (UID: \"17564146-f078-4b05-a08f-2ac7d4f1d60d\") " pod="kube-system/konnectivity-agent-ndkbc"
Sep 10 17:23:50.707385 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707253    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btlsg\" (UniqueName: \"kubernetes.io/projected/7522eaf2-4eaa-461a-9681-3df368665161-kube-api-access-btlsg\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.707385 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707285    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-run-ovn\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.707385 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707309    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-cni-bin\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.707639 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707618    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-hostroot\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.707716 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707656    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-etc-selinux\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.707716 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707680    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7067d92a-5363-42fe-a8f0-7c765931c532-host\") pod \"node-ca-wwbzr\" (UID: \"7067d92a-5363-42fe-a8f0-7c765931c532\") " pod="openshift-image-registry/node-ca-wwbzr"
Sep 10 17:23:50.707716 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707705    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lczp8\" (UniqueName: \"kubernetes.io/projected/7067d92a-5363-42fe-a8f0-7c765931c532-kube-api-access-lczp8\") pod \"node-ca-wwbzr\" (UID: \"7067d92a-5363-42fe-a8f0-7c765931c532\") " pod="openshift-image-registry/node-ca-wwbzr"
Sep 10 17:23:50.707843 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707731    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4dkx\" (UniqueName: \"kubernetes.io/projected/4d6869e8-701f-4a3f-85c6-7ee650066984-kube-api-access-n4dkx\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.707843 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707757    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:50.707843 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707758    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-etc-selinux\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.707843 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707782    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-run-multus-certs\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.707843 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707811    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-etc-kubernetes\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.707843 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707835    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d6869e8-701f-4a3f-85c6-7ee650066984-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.708073 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707859    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqn8x\" (UniqueName: \"kubernetes.io/projected/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-kube-api-access-cqn8x\") pod \"network-metrics-daemon-jk4wd\" (UID: \"3e821fe8-fb34-458c-b7d0-521a7ad7a31e\") " pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:23:50.708073 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707881    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/890f6dc7-2013-48d9-90f0-754795ed3f45-tmp\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.708073 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707907    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtq92\" (UniqueName: \"kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92\") pod \"network-check-target-9hknl\" (UID: \"c1833f06-67dc-49c7-b432-7a61d45b5319\") " pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:23:50.708073 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707931    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-var-lib-openvswitch\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.708073 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707955    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-run-openvswitch\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.708073 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.707978    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58vwq\" (UniqueName: \"kubernetes.io/projected/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-kube-api-access-58vwq\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.708073 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.708021    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d6869e8-701f-4a3f-85c6-7ee650066984-system-cni-dir\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.708073 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.708042    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7067d92a-5363-42fe-a8f0-7c765931c532-host\") pod \"node-ca-wwbzr\" (UID: \"7067d92a-5363-42fe-a8f0-7c765931c532\") " pod="openshift-image-registry/node-ca-wwbzr"
Sep 10 17:23:50.708073 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.708059    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17564146-f078-4b05-a08f-2ac7d4f1d60d-konnectivity-ca\") pod \"konnectivity-agent-ndkbc\" (UID: \"17564146-f078-4b05-a08f-2ac7d4f1d60d\") " pod="kube-system/konnectivity-agent-ndkbc"
Sep 10 17:23:50.708472 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.708083    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-var-lib-cni-bin\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.708472 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.708120    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af5509a3-a5d3-4de8-b73b-286eaed11bfc-host-slash\") pod \"iptables-alerter-vnd6t\" (UID: \"af5509a3-a5d3-4de8-b73b-286eaed11bfc\") " pod="openshift-network-operator/iptables-alerter-vnd6t"
Sep 10 17:23:50.708472 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.708145    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7067d92a-5363-42fe-a8f0-7c765931c532-serviceca\") pod \"node-ca-wwbzr\" (UID: \"7067d92a-5363-42fe-a8f0-7c765931c532\") " pod="openshift-image-registry/node-ca-wwbzr"
Sep 10 17:23:50.708472 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.708172    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-modprobe-d\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.708472 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.708206    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-cnibin\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.708472 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.708233    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-multus-conf-dir\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.708472 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.708260    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.708472 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.708280    2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 17:23:50.708472 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.708377    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d6869e8-701f-4a3f-85c6-7ee650066984-system-cni-dir\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.708880 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:50.708477    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:50.708880 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:50.708567    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret podName:ed270dc5-1f93-4739-b3b1-d5a1412d6819 nodeName:}" failed. No retries permitted until 2026-09-10 17:23:51.208545924 +0000 UTC m=+2.967072990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret") pod "global-pull-secret-syncer-bbbdj" (UID: "ed270dc5-1f93-4739-b3b1-d5a1412d6819") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:50.708976 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.708890    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17564146-f078-4b05-a08f-2ac7d4f1d60d-konnectivity-ca\") pod \"konnectivity-agent-ndkbc\" (UID: \"17564146-f078-4b05-a08f-2ac7d4f1d60d\") " pod="kube-system/konnectivity-agent-ndkbc"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.708286    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709150    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-device-dir\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709184    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-sys-fs\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709206    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-socket-dir\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709231    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81246f30-ad68-472c-aa8b-72d14634f501-hosts-file\") pod \"node-resolver-f9tg8\" (UID: \"81246f30-ad68-472c-aa8b-72d14634f501\") " pod="openshift-dns/node-resolver-f9tg8"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709253    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktcgq\" (UniqueName: \"kubernetes.io/projected/81246f30-ad68-472c-aa8b-72d14634f501-kube-api-access-ktcgq\") pod \"node-resolver-f9tg8\" (UID: \"81246f30-ad68-472c-aa8b-72d14634f501\") " pod="openshift-dns/node-resolver-f9tg8"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709276    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-etc-openvswitch\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709296    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-sysconfig\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709315    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-sysctl-conf\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709338    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7067d92a-5363-42fe-a8f0-7c765931c532-serviceca\") pod \"node-ca-wwbzr\" (UID: \"7067d92a-5363-42fe-a8f0-7c765931c532\") " pod="openshift-image-registry/node-ca-wwbzr"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709355    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af5509a3-a5d3-4de8-b73b-286eaed11bfc-iptables-alerter-script\") pod \"iptables-alerter-vnd6t\" (UID: \"af5509a3-a5d3-4de8-b73b-286eaed11bfc\") " pod="openshift-network-operator/iptables-alerter-vnd6t"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709358    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d6869e8-701f-4a3f-85c6-7ee650066984-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709376    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-run-netns\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709414    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-sys-fs\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709427    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-device-dir\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.709658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709468    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-log-socket\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709538    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-socket-dir\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709561    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81246f30-ad68-472c-aa8b-72d14634f501-tmp-dir\") pod \"node-resolver-f9tg8\" (UID: \"81246f30-ad68-472c-aa8b-72d14634f501\") " pod="openshift-dns/node-resolver-f9tg8"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709582    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-multus-cni-dir\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709611    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e992abfb-f6c8-410a-869e-7e7394580e7d-multus-daemon-config\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709630    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-slash\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709663    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-systemd\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709701    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-host\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709725    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-var-lib-cni-multus\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709767    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-var-lib-kubelet\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709791    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-run\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709812    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-lib-modules\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709838    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d6869e8-701f-4a3f-85c6-7ee650066984-cnibin\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709879    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-kubernetes\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709903    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-sysctl-d\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709925    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qt87\" (UniqueName: \"kubernetes.io/projected/e992abfb-f6c8-410a-869e-7e7394580e7d-kube-api-access-6qt87\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709950    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d6869e8-701f-4a3f-85c6-7ee650066984-cni-binary-copy\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.710410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.709984    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710005    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-sys\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710027    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-system-cni-dir\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710057    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e992abfb-f6c8-410a-869e-7e7394580e7d-cni-binary-copy\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710104    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-ovn-node-metrics-cert\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710128    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-ovnkube-script-lib\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710153    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d6869e8-701f-4a3f-85c6-7ee650066984-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710177    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-registration-dir\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710202    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-multus-socket-dir-parent\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710228    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t74wq\" (UniqueName: \"kubernetes.io/projected/af5509a3-a5d3-4de8-b73b-286eaed11bfc-kube-api-access-t74wq\") pod \"iptables-alerter-vnd6t\" (UID: \"af5509a3-a5d3-4de8-b73b-286eaed11bfc\") " pod="openshift-network-operator/iptables-alerter-vnd6t"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710253    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-systemd-units\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710276    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-cni-netd\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710302    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs\") pod \"network-metrics-daemon-jk4wd\" (UID: \"3e821fe8-fb34-458c-b7d0-521a7ad7a31e\") " pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710327    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed270dc5-1f93-4739-b3b1-d5a1412d6819-dbus\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710351    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-node-log\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710376    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-ovnkube-config\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.711223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710406    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d6869e8-701f-4a3f-85c6-7ee650066984-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710436    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed270dc5-1f93-4739-b3b1-d5a1412d6819-kubelet-config\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710461    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-tuned\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710486    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-run-k8s-cni-cncf-io\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710625    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-run-netns\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710653    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-run-systemd\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710676    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-env-overrides\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710700    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-var-lib-kubelet\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710723    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcr2k\" (UniqueName: \"kubernetes.io/projected/890f6dc7-2013-48d9-90f0-754795ed3f45-kube-api-access-mcr2k\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710745    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-kubelet\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.710824    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d6869e8-701f-4a3f-85c6-7ee650066984-cnibin\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.711274    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d6869e8-701f-4a3f-85c6-7ee650066984-cni-binary-copy\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.711330    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.711456    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d6869e8-701f-4a3f-85c6-7ee650066984-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.711519    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7522eaf2-4eaa-461a-9681-3df368665161-registration-dir\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:50.711598    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:50.712003 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:50.711665    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs podName:3e821fe8-fb34-458c-b7d0-521a7ad7a31e nodeName:}" failed. No retries permitted until 2026-09-10 17:23:51.211640351 +0000 UTC m=+2.970167419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs") pod "network-metrics-daemon-jk4wd" (UID: "3e821fe8-fb34-458c-b7d0-521a7ad7a31e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:50.712805 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.711720    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed270dc5-1f93-4739-b3b1-d5a1412d6819-dbus\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:50.712805 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.712175    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d6869e8-701f-4a3f-85c6-7ee650066984-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.712805 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.712231    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed270dc5-1f93-4739-b3b1-d5a1412d6819-kubelet-config\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:50.712982 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.712938    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17564146-f078-4b05-a08f-2ac7d4f1d60d-agent-certs\") pod \"konnectivity-agent-ndkbc\" (UID: \"17564146-f078-4b05-a08f-2ac7d4f1d60d\") " pod="kube-system/konnectivity-agent-ndkbc"
Sep 10 17:23:50.726108 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.726046    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4dkx\" (UniqueName: \"kubernetes.io/projected/4d6869e8-701f-4a3f-85c6-7ee650066984-kube-api-access-n4dkx\") pod \"multus-additional-cni-plugins-ndldz\" (UID: \"4d6869e8-701f-4a3f-85c6-7ee650066984\") " pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.727839 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.727814    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqn8x\" (UniqueName: \"kubernetes.io/projected/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-kube-api-access-cqn8x\") pod \"network-metrics-daemon-jk4wd\" (UID: \"3e821fe8-fb34-458c-b7d0-521a7ad7a31e\") " pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:23:50.729210 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.729192    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btlsg\" (UniqueName: \"kubernetes.io/projected/7522eaf2-4eaa-461a-9681-3df368665161-kube-api-access-btlsg\") pod \"aws-ebs-csi-driver-node-v9v68\" (UID: \"7522eaf2-4eaa-461a-9681-3df368665161\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.729844 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.729807    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lczp8\" (UniqueName: \"kubernetes.io/projected/7067d92a-5363-42fe-a8f0-7c765931c532-kube-api-access-lczp8\") pod \"node-ca-wwbzr\" (UID: \"7067d92a-5363-42fe-a8f0-7c765931c532\") " pod="openshift-image-registry/node-ca-wwbzr"
Sep 10 17:23:50.801421 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.801381    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:23:50.803182 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.803108    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal" event={"ID":"37cc8c335d286fe4e7fe52fe1d7f1f34","Type":"ContainerStarted","Data":"7bd45c2bda6dd6544f8de2ab911564544f83651dd9d615c08ee1e730772dbad4"}
Sep 10 17:23:50.806109 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.806085    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-90.ec2.internal" event={"ID":"9967cb0cd42cd2e177721f6d62bd6a5e","Type":"ContainerStarted","Data":"538e73f39d00912c204bf4469dddd1a34704b23a55f36e82c718aa47eebc68b9"}
Sep 10 17:23:50.811672 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811649    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-cnibin\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.811780 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811685    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-multus-conf-dir\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.811780 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811711    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.811780 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811735    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.811780 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811760    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81246f30-ad68-472c-aa8b-72d14634f501-hosts-file\") pod \"node-resolver-f9tg8\" (UID: \"81246f30-ad68-472c-aa8b-72d14634f501\") " pod="openshift-dns/node-resolver-f9tg8"
Sep 10 17:23:50.811967 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811780    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktcgq\" (UniqueName: \"kubernetes.io/projected/81246f30-ad68-472c-aa8b-72d14634f501-kube-api-access-ktcgq\") pod \"node-resolver-f9tg8\" (UID: \"81246f30-ad68-472c-aa8b-72d14634f501\") " pod="openshift-dns/node-resolver-f9tg8"
Sep 10 17:23:50.811967 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811803    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-etc-openvswitch\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.811967 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811825    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-sysconfig\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.811967 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811829    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-multus-conf-dir\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.811967 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811845    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-sysctl-conf\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.811967 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811884    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af5509a3-a5d3-4de8-b73b-286eaed11bfc-iptables-alerter-script\") pod \"iptables-alerter-vnd6t\" (UID: \"af5509a3-a5d3-4de8-b73b-286eaed11bfc\") " pod="openshift-network-operator/iptables-alerter-vnd6t"
Sep 10 17:23:50.811967 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811909    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-run-netns\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.811967 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811933    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-log-socket\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.811967 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811956    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81246f30-ad68-472c-aa8b-72d14634f501-tmp-dir\") pod \"node-resolver-f9tg8\" (UID: \"81246f30-ad68-472c-aa8b-72d14634f501\") " pod="openshift-dns/node-resolver-f9tg8"
Sep 10 17:23:50.812373 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811980    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-sysctl-conf\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.812373 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812026    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-multus-cni-dir\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.812373 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812033    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-cnibin\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.812373 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812052    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.812373 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812084    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.812373 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812137    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81246f30-ad68-472c-aa8b-72d14634f501-hosts-file\") pod \"node-resolver-f9tg8\" (UID: \"81246f30-ad68-472c-aa8b-72d14634f501\") " pod="openshift-dns/node-resolver-f9tg8"
Sep 10 17:23:50.812373 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812147    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-log-socket\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.812373 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812201    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-run-netns\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812380    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-etc-openvswitch\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812436    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-sysconfig\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.811979    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-multus-cni-dir\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812472    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e992abfb-f6c8-410a-869e-7e7394580e7d-multus-daemon-config\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812514    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-slash\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812526    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81246f30-ad68-472c-aa8b-72d14634f501-tmp-dir\") pod \"node-resolver-f9tg8\" (UID: \"81246f30-ad68-472c-aa8b-72d14634f501\") " pod="openshift-dns/node-resolver-f9tg8"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812539    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-systemd\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812563    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-host\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812588    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-var-lib-cni-multus\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812608    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-slash\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812611    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-var-lib-kubelet\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812623    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af5509a3-a5d3-4de8-b73b-286eaed11bfc-iptables-alerter-script\") pod \"iptables-alerter-vnd6t\" (UID: \"af5509a3-a5d3-4de8-b73b-286eaed11bfc\") " pod="openshift-network-operator/iptables-alerter-vnd6t"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812642    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-var-lib-kubelet\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812647    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-run\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812688    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-host\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812699    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-systemd\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812725    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-var-lib-cni-multus\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.812774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812747    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-run\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812752    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-lib-modules\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812779    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-kubernetes\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812803    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-sysctl-d\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812827    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qt87\" (UniqueName: \"kubernetes.io/projected/e992abfb-f6c8-410a-869e-7e7394580e7d-kube-api-access-6qt87\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812850    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-sys\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812891    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-system-cni-dir\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812915    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e992abfb-f6c8-410a-869e-7e7394580e7d-cni-binary-copy\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812916    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-kubernetes\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812938    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-ovn-node-metrics-cert\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812969    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-ovnkube-script-lib\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.812992    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-multus-socket-dir-parent\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813006    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-lib-modules\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813016    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t74wq\" (UniqueName: \"kubernetes.io/projected/af5509a3-a5d3-4de8-b73b-286eaed11bfc-kube-api-access-t74wq\") pod \"iptables-alerter-vnd6t\" (UID: \"af5509a3-a5d3-4de8-b73b-286eaed11bfc\") " pod="openshift-network-operator/iptables-alerter-vnd6t"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813040    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-systemd-units\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813059    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-system-cni-dir\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813062    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-cni-netd\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813103    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-cni-netd\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.813674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813108    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-node-log\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813135    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-ovnkube-config\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813162    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-tuned\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813172    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e992abfb-f6c8-410a-869e-7e7394580e7d-multus-daemon-config\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813186    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-run-k8s-cni-cncf-io\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813212    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-run-netns\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813237    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-run-systemd\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813243    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-sys\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813262    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-env-overrides\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813286    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-var-lib-kubelet\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813310    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcr2k\" (UniqueName: \"kubernetes.io/projected/890f6dc7-2013-48d9-90f0-754795ed3f45-kube-api-access-mcr2k\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813332    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-kubelet\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813347    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-sysctl-d\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813359    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-run-ovn\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813383    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-cni-bin\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813406    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-hostroot\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813451    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-run-multus-certs\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813481    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-etc-kubernetes\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.814523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813520    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/890f6dc7-2013-48d9-90f0-754795ed3f45-tmp\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813544    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtq92\" (UniqueName: \"kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92\") pod \"network-check-target-9hknl\" (UID: \"c1833f06-67dc-49c7-b432-7a61d45b5319\") " pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813570    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-var-lib-openvswitch\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813595    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-run-openvswitch\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813618    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58vwq\" (UniqueName: \"kubernetes.io/projected/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-kube-api-access-58vwq\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813647    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-var-lib-cni-bin\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813670    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af5509a3-a5d3-4de8-b73b-286eaed11bfc-host-slash\") pod \"iptables-alerter-vnd6t\" (UID: \"af5509a3-a5d3-4de8-b73b-286eaed11bfc\") " pod="openshift-network-operator/iptables-alerter-vnd6t"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813697    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-modprobe-d\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813817    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-modprobe-d\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813834    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-multus-socket-dir-parent\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813860    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-node-log\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813889    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-systemd-units\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813942    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-hostroot\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813984    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-ovnkube-script-lib\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.814292    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-ovnkube-config\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.814353    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-var-lib-openvswitch\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.814394    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-run-openvswitch\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.815327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.814637    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af5509a3-a5d3-4de8-b73b-286eaed11bfc-host-slash\") pod \"iptables-alerter-vnd6t\" (UID: \"af5509a3-a5d3-4de8-b73b-286eaed11bfc\") " pod="openshift-network-operator/iptables-alerter-vnd6t"
Sep 10 17:23:50.816123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.814638    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-run-k8s-cni-cncf-io\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.816123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.814671    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-var-lib-cni-bin\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.816123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.814722    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-run-netns\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.816123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.813545    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e992abfb-f6c8-410a-869e-7e7394580e7d-cni-binary-copy\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.816123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.814756    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-run-systemd\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.816123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.814793    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-host-run-multus-certs\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.816123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.814801    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-run-ovn\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.816123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.814831    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-cni-bin\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.816123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.814860    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e992abfb-f6c8-410a-869e-7e7394580e7d-etc-kubernetes\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.816123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.814918    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/890f6dc7-2013-48d9-90f0-754795ed3f45-var-lib-kubelet\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.816123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.814957    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-host-kubelet\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.816123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.815220    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-env-overrides\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.819613 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.819508    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/890f6dc7-2013-48d9-90f0-754795ed3f45-tmp\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.819613 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.819544    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/890f6dc7-2013-48d9-90f0-754795ed3f45-etc-tuned\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.819613 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.819558    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-ovn-node-metrics-cert\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.827619 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.827597    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktcgq\" (UniqueName: \"kubernetes.io/projected/81246f30-ad68-472c-aa8b-72d14634f501-kube-api-access-ktcgq\") pod \"node-resolver-f9tg8\" (UID: \"81246f30-ad68-472c-aa8b-72d14634f501\") " pod="openshift-dns/node-resolver-f9tg8"
Sep 10 17:23:50.827710 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:50.827623    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:23:50.827710 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:50.827647    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:23:50.827710 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:50.827670    2552 projected.go:196] Error preparing data for projected volume kube-api-access-mtq92 for pod openshift-network-diagnostics/network-check-target-9hknl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:50.827869 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:50.827743    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92 podName:c1833f06-67dc-49c7-b432-7a61d45b5319 nodeName:}" failed. No retries permitted until 2026-09-10 17:23:51.32771583 +0000 UTC m=+3.086242882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mtq92" (UniqueName: "kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92") pod "network-check-target-9hknl" (UID: "c1833f06-67dc-49c7-b432-7a61d45b5319") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:50.831955 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.831905    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t74wq\" (UniqueName: \"kubernetes.io/projected/af5509a3-a5d3-4de8-b73b-286eaed11bfc-kube-api-access-t74wq\") pod \"iptables-alerter-vnd6t\" (UID: \"af5509a3-a5d3-4de8-b73b-286eaed11bfc\") " pod="openshift-network-operator/iptables-alerter-vnd6t"
Sep 10 17:23:50.832189 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.832168    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qt87\" (UniqueName: \"kubernetes.io/projected/e992abfb-f6c8-410a-869e-7e7394580e7d-kube-api-access-6qt87\") pod \"multus-8h6xk\" (UID: \"e992abfb-f6c8-410a-869e-7e7394580e7d\") " pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.832509 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.832432    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58vwq\" (UniqueName: \"kubernetes.io/projected/a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0-kube-api-access-58vwq\") pod \"ovnkube-node-vcnth\" (UID: \"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:50.832509 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.832435    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcr2k\" (UniqueName: \"kubernetes.io/projected/890f6dc7-2013-48d9-90f0-754795ed3f45-kube-api-access-mcr2k\") pod \"tuned-b87fg\" (UID: \"890f6dc7-2013-48d9-90f0-754795ed3f45\") " pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.905131 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.905097    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ndkbc"
Sep 10 17:23:50.914936 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.914911    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68"
Sep 10 17:23:50.924252 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.924230    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wwbzr"
Sep 10 17:23:50.929989 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.929930    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ndldz"
Sep 10 17:23:50.937258 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.937236    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b87fg"
Sep 10 17:23:50.944907 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.944883    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f9tg8"
Sep 10 17:23:50.952517 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.952485    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8h6xk"
Sep 10 17:23:50.961054 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.961031    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vnd6t"
Sep 10 17:23:50.968545 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:50.968526    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:23:51.217214 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:51.217175    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs\") pod \"network-metrics-daemon-jk4wd\" (UID: \"3e821fe8-fb34-458c-b7d0-521a7ad7a31e\") " pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:23:51.217409 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:51.217228    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:51.217409 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:51.217350    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:51.217517 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:51.217426    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs podName:3e821fe8-fb34-458c-b7d0-521a7ad7a31e nodeName:}" failed. No retries permitted until 2026-09-10 17:23:52.21740572 +0000 UTC m=+3.975932794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs") pod "network-metrics-daemon-jk4wd" (UID: "3e821fe8-fb34-458c-b7d0-521a7ad7a31e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:51.217517 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:51.217350    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:51.217517 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:51.217509    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret podName:ed270dc5-1f93-4739-b3b1-d5a1412d6819 nodeName:}" failed. No retries permitted until 2026-09-10 17:23:52.217476441 +0000 UTC m=+3.976003491 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret") pod "global-pull-secret-syncer-bbbdj" (UID: "ed270dc5-1f93-4739-b3b1-d5a1412d6819") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:51.418277 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:51.418240    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtq92\" (UniqueName: \"kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92\") pod \"network-check-target-9hknl\" (UID: \"c1833f06-67dc-49c7-b432-7a61d45b5319\") " pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:23:51.418437 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:51.418392    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:23:51.418437 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:51.418411    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:23:51.418437 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:51.418423    2552 projected.go:196] Error preparing data for projected volume kube-api-access-mtq92 for pod openshift-network-diagnostics/network-check-target-9hknl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:51.418619 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:51.418486    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92 podName:c1833f06-67dc-49c7-b432-7a61d45b5319 nodeName:}" failed. No retries permitted until 2026-09-10 17:23:52.41846694 +0000 UTC m=+4.176994012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtq92" (UniqueName: "kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92") pod "network-check-target-9hknl" (UID: "c1833f06-67dc-49c7-b432-7a61d45b5319") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:51.631218 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:51.631180    2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:23:51.658824 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:51.658784    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 17:18:50 +0000 UTC" deadline="2028-06-07 22:21:21.634548706 +0000 UTC"
Sep 10 17:23:51.658824 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:51.658815    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15268h57m29.975737233s"
Sep 10 17:23:51.797670 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:51.797628    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:23:51.797670 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:51.797665    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:51.797900 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:51.797767    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:23:51.797961 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:51.797912    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:23:51.935086 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:51.935056    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6869e8_701f_4a3f_85c6_7ee650066984.slice/crio-3507dff4e602ae533b1440a69268d1f6fdf2017e327bb4656dc16fc4286e0de5 WatchSource:0}: Error finding container 3507dff4e602ae533b1440a69268d1f6fdf2017e327bb4656dc16fc4286e0de5: Status 404 returned error can't find the container with id 3507dff4e602ae533b1440a69268d1f6fdf2017e327bb4656dc16fc4286e0de5
Sep 10 17:23:51.936651 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:51.936631    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890f6dc7_2013_48d9_90f0_754795ed3f45.slice/crio-f125ecfd8b8e782de378c878a675be50bb8f5414dfb039b15481e73217a609c9 WatchSource:0}: Error finding container f125ecfd8b8e782de378c878a675be50bb8f5414dfb039b15481e73217a609c9: Status 404 returned error can't find the container with id f125ecfd8b8e782de378c878a675be50bb8f5414dfb039b15481e73217a609c9
Sep 10 17:23:51.939206 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:51.939173    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode992abfb_f6c8_410a_869e_7e7394580e7d.slice/crio-32e0e84617365cbd9f583c84c50d495ba26a6954e6faee3ebdb9d031ffe93897 WatchSource:0}: Error finding container 32e0e84617365cbd9f583c84c50d495ba26a6954e6faee3ebdb9d031ffe93897: Status 404 returned error can't find the container with id 32e0e84617365cbd9f583c84c50d495ba26a6954e6faee3ebdb9d031ffe93897
Sep 10 17:23:51.940470 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:51.940027    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5509a3_a5d3_4de8_b73b_286eaed11bfc.slice/crio-330baecbd401a9651683a0ebd11e2be07d88f2505f4284645d4e386c455f98eb WatchSource:0}: Error finding container 330baecbd401a9651683a0ebd11e2be07d88f2505f4284645d4e386c455f98eb: Status 404 returned error can't find the container with id 330baecbd401a9651683a0ebd11e2be07d88f2505f4284645d4e386c455f98eb
Sep 10 17:23:51.941067 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:51.941016    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81246f30_ad68_472c_aa8b_72d14634f501.slice/crio-a4e4affbb3b310c6b830af7a0c9f4f59b8f52bcfea155cb24d3fc1c2701ed1a6 WatchSource:0}: Error finding container a4e4affbb3b310c6b830af7a0c9f4f59b8f52bcfea155cb24d3fc1c2701ed1a6: Status 404 returned error can't find the container with id a4e4affbb3b310c6b830af7a0c9f4f59b8f52bcfea155cb24d3fc1c2701ed1a6
Sep 10 17:23:51.942603 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:51.942592    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17564146_f078_4b05_a08f_2ac7d4f1d60d.slice/crio-eb1c265c6790cdd8c70e57de1c58d30ec757e28e9e4e249f4bec482de82317b8 WatchSource:0}: Error finding container eb1c265c6790cdd8c70e57de1c58d30ec757e28e9e4e249f4bec482de82317b8: Status 404 returned error can't find the container with id eb1c265c6790cdd8c70e57de1c58d30ec757e28e9e4e249f4bec482de82317b8
Sep 10 17:23:51.944208 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:51.944188    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c88c95_a88a_45f5_8ebb_1a6e773fc3c0.slice/crio-12384d881fbe0b0ffa376d60ee718fc3b91923edd46d5b6d2de31877d991f081 WatchSource:0}: Error finding container 12384d881fbe0b0ffa376d60ee718fc3b91923edd46d5b6d2de31877d991f081: Status 404 returned error can't find the container with id 12384d881fbe0b0ffa376d60ee718fc3b91923edd46d5b6d2de31877d991f081
Sep 10 17:23:51.945609 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:51.945588    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7067d92a_5363_42fe_a8f0_7c765931c532.slice/crio-ff96e00f6cee771f994e8fdc734cc43c8d8e266f8a518d4b81680be3416ebfbb WatchSource:0}: Error finding container ff96e00f6cee771f994e8fdc734cc43c8d8e266f8a518d4b81680be3416ebfbb: Status 404 returned error can't find the container with id ff96e00f6cee771f994e8fdc734cc43c8d8e266f8a518d4b81680be3416ebfbb
Sep 10 17:23:51.947779 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:23:51.947758    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7522eaf2_4eaa_461a_9681_3df368665161.slice/crio-80b061bd2aea41b9e8a6b6dfc60ba8bd3514627add6df8a7aa35efbaf1863417 WatchSource:0}: Error finding container 80b061bd2aea41b9e8a6b6dfc60ba8bd3514627add6df8a7aa35efbaf1863417: Status 404 returned error can't find the container with id 80b061bd2aea41b9e8a6b6dfc60ba8bd3514627add6df8a7aa35efbaf1863417
Sep 10 17:23:52.070332 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.070168    2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:23:52.224412 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.224387    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:52.224547 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.224443    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs\") pod \"network-metrics-daemon-jk4wd\" (UID: \"3e821fe8-fb34-458c-b7d0-521a7ad7a31e\") " pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:23:52.224592 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:52.224548    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:52.224626 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:52.224610    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs podName:3e821fe8-fb34-458c-b7d0-521a7ad7a31e nodeName:}" failed. No retries permitted until 2026-09-10 17:23:54.224593621 +0000 UTC m=+5.983120673 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs") pod "network-metrics-daemon-jk4wd" (UID: "3e821fe8-fb34-458c-b7d0-521a7ad7a31e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:52.224667 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:52.224548    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:52.224695 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:52.224685    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret podName:ed270dc5-1f93-4739-b3b1-d5a1412d6819 nodeName:}" failed. No retries permitted until 2026-09-10 17:23:54.224671332 +0000 UTC m=+5.983198403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret") pod "global-pull-secret-syncer-bbbdj" (UID: "ed270dc5-1f93-4739-b3b1-d5a1412d6819") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:52.426206 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.426166    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtq92\" (UniqueName: \"kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92\") pod \"network-check-target-9hknl\" (UID: \"c1833f06-67dc-49c7-b432-7a61d45b5319\") " pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:23:52.426379 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:52.426338    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:23:52.426379 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:52.426361    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:23:52.426379 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:52.426371    2552 projected.go:196] Error preparing data for projected volume kube-api-access-mtq92 for pod openshift-network-diagnostics/network-check-target-9hknl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:52.426547 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:52.426419    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92 podName:c1833f06-67dc-49c7-b432-7a61d45b5319 nodeName:}" failed. No retries permitted until 2026-09-10 17:23:54.426405368 +0000 UTC m=+6.184932416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtq92" (UniqueName: "kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92") pod "network-check-target-9hknl" (UID: "c1833f06-67dc-49c7-b432-7a61d45b5319") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:52.659197 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.659155    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 17:18:50 +0000 UTC" deadline="2028-06-10 16:54:59.332964862 +0000 UTC"
Sep 10 17:23:52.659197 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.659195    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15335h31m6.67377432s"
Sep 10 17:23:52.797695 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.797620    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:23:52.797852 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:52.797761    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:23:52.830723 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.830678    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-90.ec2.internal" event={"ID":"9967cb0cd42cd2e177721f6d62bd6a5e","Type":"ContainerStarted","Data":"ef83fcb6dbdb0ea21e9f411aadfa0c5f62170d77a7811e92fa0c1b4c6a05076d"}
Sep 10 17:23:52.849930 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.849897    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68" event={"ID":"7522eaf2-4eaa-461a-9681-3df368665161","Type":"ContainerStarted","Data":"80b061bd2aea41b9e8a6b6dfc60ba8bd3514627add6df8a7aa35efbaf1863417"}
Sep 10 17:23:52.858761 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.858727    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wwbzr" event={"ID":"7067d92a-5363-42fe-a8f0-7c765931c532","Type":"ContainerStarted","Data":"ff96e00f6cee771f994e8fdc734cc43c8d8e266f8a518d4b81680be3416ebfbb"}
Sep 10 17:23:52.884698 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.884658    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth" event={"ID":"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0","Type":"ContainerStarted","Data":"12384d881fbe0b0ffa376d60ee718fc3b91923edd46d5b6d2de31877d991f081"}
Sep 10 17:23:52.893593 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.893558    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vnd6t" event={"ID":"af5509a3-a5d3-4de8-b73b-286eaed11bfc","Type":"ContainerStarted","Data":"330baecbd401a9651683a0ebd11e2be07d88f2505f4284645d4e386c455f98eb"}
Sep 10 17:23:52.907929 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.907896    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8h6xk" event={"ID":"e992abfb-f6c8-410a-869e-7e7394580e7d","Type":"ContainerStarted","Data":"32e0e84617365cbd9f583c84c50d495ba26a6954e6faee3ebdb9d031ffe93897"}
Sep 10 17:23:52.921462 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.921428    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndldz" event={"ID":"4d6869e8-701f-4a3f-85c6-7ee650066984","Type":"ContainerStarted","Data":"3507dff4e602ae533b1440a69268d1f6fdf2017e327bb4656dc16fc4286e0de5"}
Sep 10 17:23:52.935550 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.935517    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ndkbc" event={"ID":"17564146-f078-4b05-a08f-2ac7d4f1d60d","Type":"ContainerStarted","Data":"eb1c265c6790cdd8c70e57de1c58d30ec757e28e9e4e249f4bec482de82317b8"}
Sep 10 17:23:52.946968 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.946933    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f9tg8" event={"ID":"81246f30-ad68-472c-aa8b-72d14634f501","Type":"ContainerStarted","Data":"a4e4affbb3b310c6b830af7a0c9f4f59b8f52bcfea155cb24d3fc1c2701ed1a6"}
Sep 10 17:23:52.951967 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.951938    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b87fg" event={"ID":"890f6dc7-2013-48d9-90f0-754795ed3f45","Type":"ContainerStarted","Data":"f125ecfd8b8e782de378c878a675be50bb8f5414dfb039b15481e73217a609c9"}
Sep 10 17:23:52.965720 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.965687    2552 generic.go:356] "Generic (PLEG): container finished" podID="37cc8c335d286fe4e7fe52fe1d7f1f34" containerID="e26e477be6a0eb5855764c73ab4039d69c8322de4c8e6dc75c99ca7fc9227c35" exitCode=0
Sep 10 17:23:52.965841 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.965736    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal" event={"ID":"37cc8c335d286fe4e7fe52fe1d7f1f34","Type":"ContainerDied","Data":"e26e477be6a0eb5855764c73ab4039d69c8322de4c8e6dc75c99ca7fc9227c35"}
Sep 10 17:23:52.992238 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:52.991335    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-90.ec2.internal" podStartSLOduration=3.9913134 podStartE2EDuration="3.9913134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:23:52.847700293 +0000 UTC m=+4.606227365" watchObservedRunningTime="2026-09-10 17:23:52.9913134 +0000 UTC m=+4.749840471"
Sep 10 17:23:53.797170 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:53.797135    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:53.797970 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:53.797280    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:23:53.797970 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:53.797671    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:23:53.797970 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:53.797773    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:23:54.002740 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:54.002697    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal" event={"ID":"37cc8c335d286fe4e7fe52fe1d7f1f34","Type":"ContainerStarted","Data":"2c82d2b00c7b981a13b93dd008dec81bdf4dbb8c72f6149e657f24336741d5d2"}
Sep 10 17:23:54.240133 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:54.240098    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs\") pod \"network-metrics-daemon-jk4wd\" (UID: \"3e821fe8-fb34-458c-b7d0-521a7ad7a31e\") " pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:23:54.240322 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:54.240160    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:54.240322 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:54.240287    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:54.240515 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:54.240346    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret podName:ed270dc5-1f93-4739-b3b1-d5a1412d6819 nodeName:}" failed. No retries permitted until 2026-09-10 17:23:58.240328709 +0000 UTC m=+9.998855762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret") pod "global-pull-secret-syncer-bbbdj" (UID: "ed270dc5-1f93-4739-b3b1-d5a1412d6819") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:54.240771 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:54.240752    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:54.240844 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:54.240807    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs podName:3e821fe8-fb34-458c-b7d0-521a7ad7a31e nodeName:}" failed. No retries permitted until 2026-09-10 17:23:58.240791792 +0000 UTC m=+9.999318843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs") pod "network-metrics-daemon-jk4wd" (UID: "3e821fe8-fb34-458c-b7d0-521a7ad7a31e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:54.442185 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:54.442147    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtq92\" (UniqueName: \"kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92\") pod \"network-check-target-9hknl\" (UID: \"c1833f06-67dc-49c7-b432-7a61d45b5319\") " pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:23:54.442393 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:54.442370    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:23:54.442393 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:54.442392    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:23:54.442541 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:54.442405    2552 projected.go:196] Error preparing data for projected volume kube-api-access-mtq92 for pod openshift-network-diagnostics/network-check-target-9hknl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:54.442541 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:54.442461    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92 podName:c1833f06-67dc-49c7-b432-7a61d45b5319 nodeName:}" failed. No retries permitted until 2026-09-10 17:23:58.442442521 +0000 UTC m=+10.200969573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtq92" (UniqueName: "kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92") pod "network-check-target-9hknl" (UID: "c1833f06-67dc-49c7-b432-7a61d45b5319") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:54.799693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:54.798177    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:23:54.800128 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:54.799721    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:23:55.797696 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:55.797660    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:23:55.797875 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:55.797792    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:23:55.798177 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:55.798155    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:55.798272 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:55.798254    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:23:56.797710 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:56.797587    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:23:56.798159 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:56.797717    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:23:57.798505 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:57.797823    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:23:57.798505 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:57.797950    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:23:57.798505 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:57.798321    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:57.798505 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:57.798415    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:23:58.276775 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:58.276740    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs\") pod \"network-metrics-daemon-jk4wd\" (UID: \"3e821fe8-fb34-458c-b7d0-521a7ad7a31e\") " pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:23:58.276982 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:58.276801    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:58.276982 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:58.276880    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:58.276982 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:58.276921    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:58.276982 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:58.276958    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs podName:3e821fe8-fb34-458c-b7d0-521a7ad7a31e nodeName:}" failed. No retries permitted until 2026-09-10 17:24:06.276940427 +0000 UTC m=+18.035467475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs") pod "network-metrics-daemon-jk4wd" (UID: "3e821fe8-fb34-458c-b7d0-521a7ad7a31e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:58.277186 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:58.276987    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret podName:ed270dc5-1f93-4739-b3b1-d5a1412d6819 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:06.276969175 +0000 UTC m=+18.035496247 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret") pod "global-pull-secret-syncer-bbbdj" (UID: "ed270dc5-1f93-4739-b3b1-d5a1412d6819") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:58.477542 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:58.477485    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtq92\" (UniqueName: \"kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92\") pod \"network-check-target-9hknl\" (UID: \"c1833f06-67dc-49c7-b432-7a61d45b5319\") " pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:23:58.477705 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:58.477658    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:23:58.477705 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:58.477687    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:23:58.477705 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:58.477701    2552 projected.go:196] Error preparing data for projected volume kube-api-access-mtq92 for pod openshift-network-diagnostics/network-check-target-9hknl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:58.477828 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:58.477764    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92 podName:c1833f06-67dc-49c7-b432-7a61d45b5319 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:06.477744963 +0000 UTC m=+18.236272018 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtq92" (UniqueName: "kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92") pod "network-check-target-9hknl" (UID: "c1833f06-67dc-49c7-b432-7a61d45b5319") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:58.798675 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:58.798640    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:23:58.799143 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:58.798755    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:23:59.796947 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:59.796904    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:23:59.797134 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:59.797054    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:23:59.797134 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:23:59.797111    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:23:59.797257 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:23:59.797211    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:24:00.797363 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:00.797316    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:00.797871 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:00.797454    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:24:01.797270 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:01.797243    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:01.797370 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:01.797277    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:01.797370 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:01.797340    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:24:01.797724 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:01.797452    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:24:02.800452 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:02.800377    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:02.800866 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:02.800471    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:24:03.026282 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:03.026240    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68" event={"ID":"7522eaf2-4eaa-461a-9681-3df368665161","Type":"ContainerStarted","Data":"9ea9d92ab596095b27a7fad04654021194416afefce227b3f2338c4bf54c170e"}
Sep 10 17:24:03.028138 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:03.028098    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wwbzr" event={"ID":"7067d92a-5363-42fe-a8f0-7c765931c532","Type":"ContainerStarted","Data":"fea38cbe9d664d2cfeab9e6c41e4555f0c0ed8678b5cb168bced769e496273ab"}
Sep 10 17:24:03.030067 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:03.030039    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndldz" event={"ID":"4d6869e8-701f-4a3f-85c6-7ee650066984","Type":"ContainerStarted","Data":"e9625591464b9d104faff30a7033c7d4d7dc8c453b8cc6df4e77d65db7c472c0"}
Sep 10 17:24:03.031855 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:03.031797    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ndkbc" event={"ID":"17564146-f078-4b05-a08f-2ac7d4f1d60d","Type":"ContainerStarted","Data":"3b80d890080f8da863a01ea92ead832fba0147645841f03f7687c5f40e32a873"}
Sep 10 17:24:03.034560 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:03.034535    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f9tg8" event={"ID":"81246f30-ad68-472c-aa8b-72d14634f501","Type":"ContainerStarted","Data":"d7ecbceec9392fb4fd723584ef6bc39d0c0b44cfc25b7b46a3b42633fd0f078c"}
Sep 10 17:24:03.037627 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:03.037602    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b87fg" event={"ID":"890f6dc7-2013-48d9-90f0-754795ed3f45","Type":"ContainerStarted","Data":"054fa5b30b1c19332f54fc756d37dba3676d96d5e56f3ae598611afeea3f8f40"}
Sep 10 17:24:03.044609 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:03.044545    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-90.ec2.internal" podStartSLOduration=14.044526367 podStartE2EDuration="14.044526367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:23:54.019485386 +0000 UTC m=+5.778012480" watchObservedRunningTime="2026-09-10 17:24:03.044526367 +0000 UTC m=+14.803053441"
Sep 10 17:24:03.045340 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:03.045266    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wwbzr" podStartSLOduration=4.107187938 podStartE2EDuration="14.045254124s" totalImagesPullingTime="9.938066186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:03.043585641 +0000 UTC m=+14.802112706" watchObservedRunningTime="2026-09-10 17:24:03.045254124 +0000 UTC m=+14.803781194"
Sep 10 17:24:03.092584 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:03.092527    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b87fg" podStartSLOduration=3.906247482 podStartE2EDuration="14.092509908s" totalImagesPullingTime="10.186262426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:03.091964312 +0000 UTC m=+14.850491385" watchObservedRunningTime="2026-09-10 17:24:03.092509908 +0000 UTC m=+14.851036982"
Sep 10 17:24:03.161139 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:03.161087    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f9tg8" podStartSLOduration=4.218636871 podStartE2EDuration="14.161067635s" totalImagesPullingTime="9.942430764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:03.113350481 +0000 UTC m=+14.871877550" watchObservedRunningTime="2026-09-10 17:24:03.161067635 +0000 UTC m=+14.919594703"
Sep 10 17:24:03.161435 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:03.161412    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ndkbc" podStartSLOduration=4.245761734 podStartE2EDuration="14.161405225s" totalImagesPullingTime="9.915643491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:03.161194518 +0000 UTC m=+14.919721589" watchObservedRunningTime="2026-09-10 17:24:03.161405225 +0000 UTC m=+14.919932295"
Sep 10 17:24:03.797017 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:03.796984    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:03.797189 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:03.796998    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:03.797189 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:03.797117    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:24:03.797189 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:03.797151    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:24:04.042260 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:04.042156    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vnd6t" event={"ID":"af5509a3-a5d3-4de8-b73b-286eaed11bfc","Type":"ContainerStarted","Data":"98f66eff37c2f9078367f424edcff2a9672a94f4f697d3e23a8863e2201adabb"}
Sep 10 17:24:04.058317 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:04.058262    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vnd6t" podStartSLOduration=5.108989358 podStartE2EDuration="15.058246068s" totalImagesPullingTime="9.94925671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:04.058145798 +0000 UTC m=+15.816672869" watchObservedRunningTime="2026-09-10 17:24:04.058246068 +0000 UTC m=+15.816773139"
Sep 10 17:24:04.797359 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:04.797327    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:04.797546 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:04.797451    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:24:05.797221 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:05.797174    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:05.797803 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:05.797188    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:05.797803 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:05.797331    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:24:05.797803 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:05.797369    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:24:06.048807 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:06.048715    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d6869e8-701f-4a3f-85c6-7ee650066984" containerID="e9625591464b9d104faff30a7033c7d4d7dc8c453b8cc6df4e77d65db7c472c0" exitCode=0
Sep 10 17:24:06.048807 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:06.048768    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndldz" event={"ID":"4d6869e8-701f-4a3f-85c6-7ee650066984","Type":"ContainerDied","Data":"e9625591464b9d104faff30a7033c7d4d7dc8c453b8cc6df4e77d65db7c472c0"}
Sep 10 17:24:06.335768 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:06.335733    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs\") pod \"network-metrics-daemon-jk4wd\" (UID: \"3e821fe8-fb34-458c-b7d0-521a7ad7a31e\") " pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:06.335933 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:06.335800    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:06.335933 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:06.335902    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:24:06.335933 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:06.335928    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:24:06.336083 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:06.335979    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs podName:3e821fe8-fb34-458c-b7d0-521a7ad7a31e nodeName:}" failed. No retries permitted until 2026-09-10 17:24:22.335961352 +0000 UTC m=+34.094488402 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs") pod "network-metrics-daemon-jk4wd" (UID: "3e821fe8-fb34-458c-b7d0-521a7ad7a31e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:24:06.336083 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:06.335999    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret podName:ed270dc5-1f93-4739-b3b1-d5a1412d6819 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:22.335989622 +0000 UTC m=+34.094516670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret") pod "global-pull-secret-syncer-bbbdj" (UID: "ed270dc5-1f93-4739-b3b1-d5a1412d6819") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:24:06.538034 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:06.537994    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtq92\" (UniqueName: \"kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92\") pod \"network-check-target-9hknl\" (UID: \"c1833f06-67dc-49c7-b432-7a61d45b5319\") " pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:06.538178 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:06.538164    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:24:06.538238 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:06.538187    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:24:06.538238 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:06.538200    2552 projected.go:196] Error preparing data for projected volume kube-api-access-mtq92 for pod openshift-network-diagnostics/network-check-target-9hknl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:24:06.538314 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:06.538252    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92 podName:c1833f06-67dc-49c7-b432-7a61d45b5319 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:22.53823934 +0000 UTC m=+34.296766388 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtq92" (UniqueName: "kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92") pod "network-check-target-9hknl" (UID: "c1833f06-67dc-49c7-b432-7a61d45b5319") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:24:06.798018 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:06.797937    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:06.798440 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:06.798057    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:24:07.797650 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:07.797613    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:07.797837 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:07.797655    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:07.797837 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:07.797750    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:24:07.797943 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:07.797885    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:24:07.892771 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:07.892733    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ndkbc"
Sep 10 17:24:07.893350 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:07.893329    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ndkbc"
Sep 10 17:24:08.798381 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:08.798341    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:08.798561 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:08.798466    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:24:09.797044 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:09.797019    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:09.797044 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:09.797034    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:09.797397 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:09.797149    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:24:09.797397 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:09.797263    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:24:09.938862 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:09.938812    2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 17:24:10.060562 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:10.060531    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68" event={"ID":"7522eaf2-4eaa-461a-9681-3df368665161","Type":"ContainerStarted","Data":"ec872a23012e60a192170fa4bdfcf6f1d8ef08cb44aa7ac67934780193ffcd64"}
Sep 10 17:24:10.063644 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:10.063599    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth" event={"ID":"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0","Type":"ContainerStarted","Data":"9d6557274d5852345e889450361a0e95deb5629ff9cbc481219edeca6dee84d5"}
Sep 10 17:24:10.063764 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:10.063683    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth" event={"ID":"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0","Type":"ContainerStarted","Data":"a5d93c05a0852a46823e8f9294d546555509442fec4d5c63c782115275aebabf"}
Sep 10 17:24:10.063764 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:10.063734    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth" event={"ID":"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0","Type":"ContainerStarted","Data":"4594aafd224e9fc1279619788416a267cd901ead52eabad643cf65ea1ee41961"}
Sep 10 17:24:10.065766 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:10.065738    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8h6xk" event={"ID":"e992abfb-f6c8-410a-869e-7e7394580e7d","Type":"ContainerStarted","Data":"a80fb88fbd4f210158dc55c78b8ee9c3451f9d95f6f820d979608d9ed2e0dd7a"}
Sep 10 17:24:10.085664 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:10.085614    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8h6xk" podStartSLOduration=3.234608018 podStartE2EDuration="21.085596939s" totalImagesPullingTime="17.850988921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:10.085429937 +0000 UTC m=+21.843957008" watchObservedRunningTime="2026-09-10 17:24:10.085596939 +0000 UTC m=+21.844124012"
Sep 10 17:24:10.654391 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:10.654286    2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T17:24:09.93883387Z","UUID":"785a7ba7-5e97-4959-94da-00bc04bf4d19","Handler":null,"Name":"","Endpoint":""}
Sep 10 17:24:10.655949 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:10.655928    2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 17:24:10.656085 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:10.655960    2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 17:24:10.797761 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:10.797611    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:10.797761 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:10.797741    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:24:11.072585 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:11.072486    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth" event={"ID":"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0","Type":"ContainerStarted","Data":"de55356c1b0fbf4eba43b8a783dce1e92a94344c5c481004ff264d7eb675af3e"}
Sep 10 17:24:11.072737 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:11.072590    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth" event={"ID":"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0","Type":"ContainerStarted","Data":"b757d211fc0ecb7ce38963bb2daccd985bb788c10fddee78810e708ccf186c7d"}
Sep 10 17:24:11.072737 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:11.072636    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth" event={"ID":"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0","Type":"ContainerStarted","Data":"196fbcf67677690fe597099e37036a29a2e3f845926c5bfe548945d82491038d"}
Sep 10 17:24:11.797582 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:11.797476    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:11.797747 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:11.797478    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:11.797747 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:11.797644    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:24:11.798203 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:11.797747    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:24:12.077065 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:12.076971    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68" event={"ID":"7522eaf2-4eaa-461a-9681-3df368665161","Type":"ContainerStarted","Data":"8ccb118889546a2211c7c27077df5d79fffa1edd9d50211c1ee688cd3f155c14"}
Sep 10 17:24:12.797289 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:12.797210    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:12.797458 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:12.797330    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:24:13.082414 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:13.082335    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth" event={"ID":"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0","Type":"ContainerStarted","Data":"0da54714f9578f2b206c8093ce5bfeb43a3b0effb551a771b9e1200856780ebc"}
Sep 10 17:24:13.761048 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:13.761014    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ndkbc"
Sep 10 17:24:13.761232 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:13.761156    2552 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 17:24:13.761888 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:13.761857    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ndkbc"
Sep 10 17:24:13.777566 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:13.777512    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9v68" podStartSLOduration=5.943745365 podStartE2EDuration="24.777478689s" totalImagesPullingTime="18.833733324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:12.106904073 +0000 UTC m=+23.865431142" watchObservedRunningTime="2026-09-10 17:24:13.777478689 +0000 UTC m=+25.536005760"
Sep 10 17:24:13.797392 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:13.797366    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:13.797538 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:13.797520    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:13.797691 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:13.797516    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:24:13.797691 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:13.797665    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:24:14.797235 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:14.797051    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:14.797679 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:14.797341    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:24:15.090281 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:15.090248    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth" event={"ID":"a4c88c95-a88a-45f5-8ebb-1a6e773fc3c0","Type":"ContainerStarted","Data":"c0c6a3b71b6cbfff68cc1c3b9545c57f2f7d8c8ec9f7d26f0a2b2f665c7a9b10"}
Sep 10 17:24:15.090568 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:15.090476    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:24:15.090568 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:15.090563    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:24:15.090683 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:15.090597    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:24:15.092343 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:15.092318    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d6869e8-701f-4a3f-85c6-7ee650066984" containerID="b55737e0d53cebcf9a15e66b6c6fc7c8f0f92ec16590916524e8a3001eefe787" exitCode=0
Sep 10 17:24:15.092483 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:15.092354    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndldz" event={"ID":"4d6869e8-701f-4a3f-85c6-7ee650066984","Type":"ContainerDied","Data":"b55737e0d53cebcf9a15e66b6c6fc7c8f0f92ec16590916524e8a3001eefe787"}
Sep 10 17:24:15.106109 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:15.106086    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:24:15.106241 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:15.106223    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:24:15.124332 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:15.124294    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth" podStartSLOduration=8.278809147 podStartE2EDuration="26.124281707s" totalImagesPullingTime="17.84547256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:15.122726279 +0000 UTC m=+26.881253348" watchObservedRunningTime="2026-09-10 17:24:15.124281707 +0000 UTC m=+26.882808776"
Sep 10 17:24:15.797119 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:15.797082    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:15.797119 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:15.797111    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:15.797768 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:15.797223    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:24:15.797768 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:15.797357    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:24:16.518660 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:16.518621    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9hknl"]
Sep 10 17:24:16.518842 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:16.518733    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:16.518842 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:16.518809    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:24:16.519241 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:16.519219    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bbbdj"]
Sep 10 17:24:16.519447 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:16.519309    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:16.519447 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:16.519402    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:24:16.520619 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:16.520600    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jk4wd"]
Sep 10 17:24:16.520709 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:16.520664    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:16.520746 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:16.520729    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:24:17.099395 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:17.099360    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d6869e8-701f-4a3f-85c6-7ee650066984" containerID="eae66641d2064c3c28de3acb29182b8b11179a8189ffb29be26b7acd310fda28" exitCode=0
Sep 10 17:24:17.100079 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:17.099433    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndldz" event={"ID":"4d6869e8-701f-4a3f-85c6-7ee650066984","Type":"ContainerDied","Data":"eae66641d2064c3c28de3acb29182b8b11179a8189ffb29be26b7acd310fda28"}
Sep 10 17:24:17.797813 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:17.797784    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:17.797993 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:17.797897    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:24:17.797993 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:17.797976    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:17.798093 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:17.798067    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:24:17.798163 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:17.798113    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:17.798212 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:17.798189    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:24:18.104445 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:18.104420    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndldz" event={"ID":"4d6869e8-701f-4a3f-85c6-7ee650066984","Type":"ContainerStarted","Data":"63b4d6141ef4925f257eaf9745a2a60d7ca1666136844f4c0baca730c68017d0"}
Sep 10 17:24:19.109824 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:19.109791    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d6869e8-701f-4a3f-85c6-7ee650066984" containerID="63b4d6141ef4925f257eaf9745a2a60d7ca1666136844f4c0baca730c68017d0" exitCode=0
Sep 10 17:24:19.110227 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:19.109835    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndldz" event={"ID":"4d6869e8-701f-4a3f-85c6-7ee650066984","Type":"ContainerDied","Data":"63b4d6141ef4925f257eaf9745a2a60d7ca1666136844f4c0baca730c68017d0"}
Sep 10 17:24:19.797412 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:19.797234    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:19.797800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:19.797280    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:19.797800 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:19.797485    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:24:19.797800 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:19.797589    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:24:19.797800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:19.797278    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:19.797800 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:19.797661    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:24:21.797095 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:21.797055    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:21.797095 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:21.797085    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:21.797095 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:21.797105    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:21.797700 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:21.797201    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbbdj" podUID="ed270dc5-1f93-4739-b3b1-d5a1412d6819"
Sep 10 17:24:21.797700 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:21.797351    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jk4wd" podUID="3e821fe8-fb34-458c-b7d0-521a7ad7a31e"
Sep 10 17:24:21.797700 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:21.797426    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hknl" podUID="c1833f06-67dc-49c7-b432-7a61d45b5319"
Sep 10 17:24:22.355933 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.355895    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs\") pod \"network-metrics-daemon-jk4wd\" (UID: \"3e821fe8-fb34-458c-b7d0-521a7ad7a31e\") " pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:22.355933 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.355939    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:22.356175 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:22.356041    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:24:22.356175 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:22.356061    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:24:22.356175 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:22.356090    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret podName:ed270dc5-1f93-4739-b3b1-d5a1412d6819 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:54.356077602 +0000 UTC m=+66.114604650 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret") pod "global-pull-secret-syncer-bbbdj" (UID: "ed270dc5-1f93-4739-b3b1-d5a1412d6819") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:24:22.356175 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:22.356122    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs podName:3e821fe8-fb34-458c-b7d0-521a7ad7a31e nodeName:}" failed. No retries permitted until 2026-09-10 17:24:54.356104716 +0000 UTC m=+66.114631778 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs") pod "network-metrics-daemon-jk4wd" (UID: "3e821fe8-fb34-458c-b7d0-521a7ad7a31e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:24:22.557333 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.557302    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtq92\" (UniqueName: \"kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92\") pod \"network-check-target-9hknl\" (UID: \"c1833f06-67dc-49c7-b432-7a61d45b5319\") " pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:22.557555 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:22.557489    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:24:22.557555 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:22.557529    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:24:22.557555 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:22.557543    2552 projected.go:196] Error preparing data for projected volume kube-api-access-mtq92 for pod openshift-network-diagnostics/network-check-target-9hknl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:24:22.557739 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:22.557607    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92 podName:c1833f06-67dc-49c7-b432-7a61d45b5319 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:54.557586696 +0000 UTC m=+66.316113762 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtq92" (UniqueName: "kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92") pod "network-check-target-9hknl" (UID: "c1833f06-67dc-49c7-b432-7a61d45b5319") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:24:22.579922 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.579885    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-90.ec2.internal" event="NodeReady"
Sep 10 17:24:22.580078 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.580044    2552 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 17:24:22.625246 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.625214    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64f498f4fc-7k7bl"]
Sep 10 17:24:22.656686 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.656658    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64f498f4fc-7k7bl"]
Sep 10 17:24:22.656686 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.656693    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-77x6z"]
Sep 10 17:24:22.656854 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.656778    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.659688 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.659665    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 17:24:22.659945 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.659930    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 17:24:22.660488 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.660467    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 17:24:22.661439 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.661420    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kwhpq\""
Sep 10 17:24:22.665952 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.665085    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 17:24:22.669309 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.669289    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-55x6q"]
Sep 10 17:24:22.669432 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.669417    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:22.672204 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.672187    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 17:24:22.672744 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.672731    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 17:24:22.672816 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.672746    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vn8bp\""
Sep 10 17:24:22.691298 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.691277    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-55x6q"]
Sep 10 17:24:22.691298 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.691306    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-77x6z"]
Sep 10 17:24:22.691486 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.691382    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-55x6q"
Sep 10 17:24:22.693912 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.693889    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 17:24:22.694005 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.693958    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 17:24:22.694094 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.694080    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dfch6\""
Sep 10 17:24:22.694283 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.694271    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 17:24:22.759280 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.759197    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-certificates\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.759280 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.759244    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fff24ddc-90d7-4565-8f85-27c6f028cca4-cert\") pod \"ingress-canary-55x6q\" (UID: \"fff24ddc-90d7-4565-8f85-27c6f028cca4\") " pod="openshift-ingress-canary/ingress-canary-55x6q"
Sep 10 17:24:22.759280 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.759279    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b1670d-12c0-4614-a86e-22c5498e1274-config-volume\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:22.759562 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.759318    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47b034f8-d52d-40ca-9c28-4e2931eb61b3-trusted-ca\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.759562 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.759386    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/47b034f8-d52d-40ca-9c28-4e2931eb61b3-image-registry-private-configuration\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.759562 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.759471    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/47b034f8-d52d-40ca-9c28-4e2931eb61b3-installation-pull-secrets\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.759562 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.759528    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knkm8\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-kube-api-access-knkm8\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.759562 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.759558    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:22.759801 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.759587    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-bound-sa-token\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.759801 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.759630    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq628\" (UniqueName: \"kubernetes.io/projected/fff24ddc-90d7-4565-8f85-27c6f028cca4-kube-api-access-vq628\") pod \"ingress-canary-55x6q\" (UID: \"fff24ddc-90d7-4565-8f85-27c6f028cca4\") " pod="openshift-ingress-canary/ingress-canary-55x6q"
Sep 10 17:24:22.759801 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.759658    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b1670d-12c0-4614-a86e-22c5498e1274-tmp-dir\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:22.759801 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.759745    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.759801 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.759784    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/47b034f8-d52d-40ca-9c28-4e2931eb61b3-ca-trust-extracted\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.759984 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.759822    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxls5\" (UniqueName: \"kubernetes.io/projected/a3b1670d-12c0-4614-a86e-22c5498e1274-kube-api-access-hxls5\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:22.860714 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.860684    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/47b034f8-d52d-40ca-9c28-4e2931eb61b3-ca-trust-extracted\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.860714 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.860719    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxls5\" (UniqueName: \"kubernetes.io/projected/a3b1670d-12c0-4614-a86e-22c5498e1274-kube-api-access-hxls5\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:22.861306 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.860736    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-certificates\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.861306 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.860760    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fff24ddc-90d7-4565-8f85-27c6f028cca4-cert\") pod \"ingress-canary-55x6q\" (UID: \"fff24ddc-90d7-4565-8f85-27c6f028cca4\") " pod="openshift-ingress-canary/ingress-canary-55x6q"
Sep 10 17:24:22.861306 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.860784    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b1670d-12c0-4614-a86e-22c5498e1274-config-volume\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:22.861306 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.860816    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47b034f8-d52d-40ca-9c28-4e2931eb61b3-trusted-ca\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.861306 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.860856    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/47b034f8-d52d-40ca-9c28-4e2931eb61b3-image-registry-private-configuration\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.861306 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.861158    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/47b034f8-d52d-40ca-9c28-4e2931eb61b3-installation-pull-secrets\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.861306 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.861198    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/47b034f8-d52d-40ca-9c28-4e2931eb61b3-ca-trust-extracted\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.861306 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.861208    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knkm8\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-kube-api-access-knkm8\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.861306 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.861237    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:22.861306 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.861263    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-bound-sa-token\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.861306 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.861288    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq628\" (UniqueName: \"kubernetes.io/projected/fff24ddc-90d7-4565-8f85-27c6f028cca4-kube-api-access-vq628\") pod \"ingress-canary-55x6q\" (UID: \"fff24ddc-90d7-4565-8f85-27c6f028cca4\") " pod="openshift-ingress-canary/ingress-canary-55x6q"
Sep 10 17:24:22.861306 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.861314    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b1670d-12c0-4614-a86e-22c5498e1274-tmp-dir\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:22.861945 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.861361    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.861945 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.861461    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-certificates\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.861945 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:22.861469    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:24:22.861945 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.861516    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b1670d-12c0-4614-a86e-22c5498e1274-config-volume\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:22.861945 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:22.861852    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:24:22.861945 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:22.861922    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls podName:a3b1670d-12c0-4614-a86e-22c5498e1274 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:23.361901992 +0000 UTC m=+35.120429060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls") pod "dns-default-77x6z" (UID: "a3b1670d-12c0-4614-a86e-22c5498e1274") : secret "dns-default-metrics-tls" not found
Sep 10 17:24:22.862246 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.862097    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b1670d-12c0-4614-a86e-22c5498e1274-tmp-dir\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:22.862246 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:22.862154    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f498f4fc-7k7bl: secret "image-registry-tls" not found
Sep 10 17:24:22.862246 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.862162    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47b034f8-d52d-40ca-9c28-4e2931eb61b3-trusted-ca\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.862246 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:22.862215    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls podName:47b034f8-d52d-40ca-9c28-4e2931eb61b3 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:23.362198507 +0000 UTC m=+35.120725575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls") pod "image-registry-64f498f4fc-7k7bl" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3") : secret "image-registry-tls" not found
Sep 10 17:24:22.865043 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.865015    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/47b034f8-d52d-40ca-9c28-4e2931eb61b3-image-registry-private-configuration\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.865043 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.865023    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fff24ddc-90d7-4565-8f85-27c6f028cca4-cert\") pod \"ingress-canary-55x6q\" (UID: \"fff24ddc-90d7-4565-8f85-27c6f028cca4\") " pod="openshift-ingress-canary/ingress-canary-55x6q"
Sep 10 17:24:22.865043 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.865018    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/47b034f8-d52d-40ca-9c28-4e2931eb61b3-installation-pull-secrets\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.872115 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.872088    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-bound-sa-token\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.872223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.872192    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxls5\" (UniqueName: \"kubernetes.io/projected/a3b1670d-12c0-4614-a86e-22c5498e1274-kube-api-access-hxls5\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:22.873414 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.873395    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knkm8\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-kube-api-access-knkm8\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:22.873485 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.873456    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq628\" (UniqueName: \"kubernetes.io/projected/fff24ddc-90d7-4565-8f85-27c6f028cca4-kube-api-access-vq628\") pod \"ingress-canary-55x6q\" (UID: \"fff24ddc-90d7-4565-8f85-27c6f028cca4\") " pod="openshift-ingress-canary/ingress-canary-55x6q"
Sep 10 17:24:22.999829 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:22.999795    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-55x6q"
Sep 10 17:24:23.327058 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:23.326974    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-55x6q"]
Sep 10 17:24:23.331222 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:24:23.331193    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff24ddc_90d7_4565_8f85_27c6f028cca4.slice/crio-8abb22742f8c1f17bbfa8010155b613ea2dcf2785b7a5d930fbdedd696027355 WatchSource:0}: Error finding container 8abb22742f8c1f17bbfa8010155b613ea2dcf2785b7a5d930fbdedd696027355: Status 404 returned error can't find the container with id 8abb22742f8c1f17bbfa8010155b613ea2dcf2785b7a5d930fbdedd696027355
Sep 10 17:24:23.366976 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:23.366943    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:23.367116 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:23.367083    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:23.367116 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:23.367093    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:24:23.367222 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:23.367157    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls podName:a3b1670d-12c0-4614-a86e-22c5498e1274 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:24.367135258 +0000 UTC m=+36.125662327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls") pod "dns-default-77x6z" (UID: "a3b1670d-12c0-4614-a86e-22c5498e1274") : secret "dns-default-metrics-tls" not found
Sep 10 17:24:23.367222 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:23.367167    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:24:23.367222 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:23.367177    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f498f4fc-7k7bl: secret "image-registry-tls" not found
Sep 10 17:24:23.367222 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:23.367211    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls podName:47b034f8-d52d-40ca-9c28-4e2931eb61b3 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:24.367200691 +0000 UTC m=+36.125727755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls") pod "image-registry-64f498f4fc-7k7bl" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3") : secret "image-registry-tls" not found
Sep 10 17:24:23.797124 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:23.797087    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:23.797308 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:23.797209    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:23.797308 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:23.797246    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:23.801117 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:23.801093    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v58r5\""
Sep 10 17:24:23.801117 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:23.801111    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 17:24:23.801300 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:23.801121    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 17:24:23.801300 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:23.801144    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qjn9k\""
Sep 10 17:24:23.801444 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:23.801428    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 17:24:23.801676 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:23.801662    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 17:24:24.123586 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:24.123336    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-55x6q" event={"ID":"fff24ddc-90d7-4565-8f85-27c6f028cca4","Type":"ContainerStarted","Data":"8abb22742f8c1f17bbfa8010155b613ea2dcf2785b7a5d930fbdedd696027355"}
Sep 10 17:24:24.376074 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:24.375990    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:24.376245 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:24.376070    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:24.376245 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:24.376116    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:24:24.376245 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:24.376178    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls podName:a3b1670d-12c0-4614-a86e-22c5498e1274 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:26.376161 +0000 UTC m=+38.134688048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls") pod "dns-default-77x6z" (UID: "a3b1670d-12c0-4614-a86e-22c5498e1274") : secret "dns-default-metrics-tls" not found
Sep 10 17:24:24.376245 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:24.376177    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:24:24.376245 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:24.376196    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f498f4fc-7k7bl: secret "image-registry-tls" not found
Sep 10 17:24:24.376245 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:24.376219    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls podName:47b034f8-d52d-40ca-9c28-4e2931eb61b3 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:26.376212846 +0000 UTC m=+38.134739894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls") pod "image-registry-64f498f4fc-7k7bl" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3") : secret "image-registry-tls" not found
Sep 10 17:24:26.393182 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:26.393144    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:26.393661 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:26.393210    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:26.393661 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:26.393316    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:24:26.393661 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:26.393342    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f498f4fc-7k7bl: secret "image-registry-tls" not found
Sep 10 17:24:26.393661 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:26.393409    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls podName:47b034f8-d52d-40ca-9c28-4e2931eb61b3 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:30.393391725 +0000 UTC m=+42.151918785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls") pod "image-registry-64f498f4fc-7k7bl" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3") : secret "image-registry-tls" not found
Sep 10 17:24:26.393661 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:26.393313    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:24:26.393661 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:26.393453    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls podName:a3b1670d-12c0-4614-a86e-22c5498e1274 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:30.393442456 +0000 UTC m=+42.151969513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls") pod "dns-default-77x6z" (UID: "a3b1670d-12c0-4614-a86e-22c5498e1274") : secret "dns-default-metrics-tls" not found
Sep 10 17:24:27.134178 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:27.134142    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndldz" event={"ID":"4d6869e8-701f-4a3f-85c6-7ee650066984","Type":"ContainerStarted","Data":"bc4462b0e2c7791728490336594d73a8d7d9686311f9e0b07ebd59f482869ea8"}
Sep 10 17:24:28.139628 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:28.139596    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d6869e8-701f-4a3f-85c6-7ee650066984" containerID="bc4462b0e2c7791728490336594d73a8d7d9686311f9e0b07ebd59f482869ea8" exitCode=0
Sep 10 17:24:28.140331 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:28.139668    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndldz" event={"ID":"4d6869e8-701f-4a3f-85c6-7ee650066984","Type":"ContainerDied","Data":"bc4462b0e2c7791728490336594d73a8d7d9686311f9e0b07ebd59f482869ea8"}
Sep 10 17:24:28.141142 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:28.141112    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-55x6q" event={"ID":"fff24ddc-90d7-4565-8f85-27c6f028cca4","Type":"ContainerStarted","Data":"c4eece66056c5a1eaad3934ed1c1b1320186a601032a53cb7d78d4810a50f9bd"}
Sep 10 17:24:28.179584 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:28.179526    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-55x6q" podStartSLOduration=2.239405155 podStartE2EDuration="6.179474443s" totalImagesPullingTime="3.940069288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:28.178583561 +0000 UTC m=+39.937110632" watchObservedRunningTime="2026-09-10 17:24:28.179474443 +0000 UTC m=+39.938001519"
Sep 10 17:24:29.146576 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:29.146541    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d6869e8-701f-4a3f-85c6-7ee650066984" containerID="5213bd906c6dc67e4d5e6c13ba8564d55e0212baad3a8e49b423dda7c519bad6" exitCode=0
Sep 10 17:24:29.147039 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:29.146609    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndldz" event={"ID":"4d6869e8-701f-4a3f-85c6-7ee650066984","Type":"ContainerDied","Data":"5213bd906c6dc67e4d5e6c13ba8564d55e0212baad3a8e49b423dda7c519bad6"}
Sep 10 17:24:30.151883 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:30.151695    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndldz" event={"ID":"4d6869e8-701f-4a3f-85c6-7ee650066984","Type":"ContainerStarted","Data":"0f956610a2dfa0c47a134cf6b2b9f492e2cc8a017cf84f8659b6fd7e77de2849"}
Sep 10 17:24:30.177296 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:30.177251    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ndldz" podStartSLOduration=12.921969217000001 podStartE2EDuration="41.177234951s" totalImagesPullingTime="28.255265734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:30.175861811 +0000 UTC m=+41.934388881" watchObservedRunningTime="2026-09-10 17:24:30.177234951 +0000 UTC m=+41.935762020"
Sep 10 17:24:30.423429 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:30.423306    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:30.423429 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:30.423364    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:30.423630 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:30.423454    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:24:30.423630 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:30.423485    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:24:30.423630 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:30.423520    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f498f4fc-7k7bl: secret "image-registry-tls" not found
Sep 10 17:24:30.423630 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:30.423540    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls podName:a3b1670d-12c0-4614-a86e-22c5498e1274 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:38.423516929 +0000 UTC m=+50.182043985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls") pod "dns-default-77x6z" (UID: "a3b1670d-12c0-4614-a86e-22c5498e1274") : secret "dns-default-metrics-tls" not found
Sep 10 17:24:30.423630 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:30.423558    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls podName:47b034f8-d52d-40ca-9c28-4e2931eb61b3 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:38.423546863 +0000 UTC m=+50.182073915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls") pod "image-registry-64f498f4fc-7k7bl" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3") : secret "image-registry-tls" not found
Sep 10 17:24:38.486468 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:38.486419    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:38.486960 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:38.486510    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:38.486960 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:38.486591    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:24:38.486960 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:38.486593    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:24:38.486960 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:38.486614    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f498f4fc-7k7bl: secret "image-registry-tls" not found
Sep 10 17:24:38.486960 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:38.486643    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls podName:a3b1670d-12c0-4614-a86e-22c5498e1274 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:54.486629857 +0000 UTC m=+66.245156906 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls") pod "dns-default-77x6z" (UID: "a3b1670d-12c0-4614-a86e-22c5498e1274") : secret "dns-default-metrics-tls" not found
Sep 10 17:24:38.486960 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:38.486669    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls podName:47b034f8-d52d-40ca-9c28-4e2931eb61b3 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:54.486654889 +0000 UTC m=+66.245181940 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls") pod "image-registry-64f498f4fc-7k7bl" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3") : secret "image-registry-tls" not found
Sep 10 17:24:47.109730 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:47.109701    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcnth"
Sep 10 17:24:54.398876 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.398837    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs\") pod \"network-metrics-daemon-jk4wd\" (UID: \"3e821fe8-fb34-458c-b7d0-521a7ad7a31e\") " pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:24:54.399269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.398894    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:54.401797 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.401777    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 17:24:54.401844 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.401802    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 17:24:54.410040 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:54.410017    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 17:24:54.410106 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:54.410096    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs podName:3e821fe8-fb34-458c-b7d0-521a7ad7a31e nodeName:}" failed. No retries permitted until 2026-09-10 17:25:58.410078687 +0000 UTC m=+130.168605738 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs") pod "network-metrics-daemon-jk4wd" (UID: "3e821fe8-fb34-458c-b7d0-521a7ad7a31e") : secret "metrics-daemon-secret" not found
Sep 10 17:24:54.413375 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.413348    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed270dc5-1f93-4739-b3b1-d5a1412d6819-original-pull-secret\") pod \"global-pull-secret-syncer-bbbdj\" (UID: \"ed270dc5-1f93-4739-b3b1-d5a1412d6819\") " pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:54.418289 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.418270    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbbdj"
Sep 10 17:24:54.499935 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.499902    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:24:54.500098 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.499967    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:24:54.500098 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:54.500063    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:24:54.500200 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:54.500137    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls podName:a3b1670d-12c0-4614-a86e-22c5498e1274 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:26.500118094 +0000 UTC m=+98.258645142 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls") pod "dns-default-77x6z" (UID: "a3b1670d-12c0-4614-a86e-22c5498e1274") : secret "dns-default-metrics-tls" not found
Sep 10 17:24:54.500200 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:54.500067    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:24:54.500200 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:54.500177    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f498f4fc-7k7bl: secret "image-registry-tls" not found
Sep 10 17:24:54.500330 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:24:54.500247    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls podName:47b034f8-d52d-40ca-9c28-4e2931eb61b3 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:26.500230845 +0000 UTC m=+98.258757895 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls") pod "image-registry-64f498f4fc-7k7bl" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3") : secret "image-registry-tls" not found
Sep 10 17:24:54.600636 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.600591    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtq92\" (UniqueName: \"kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92\") pod \"network-check-target-9hknl\" (UID: \"c1833f06-67dc-49c7-b432-7a61d45b5319\") " pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:54.603642 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.603621    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 17:24:54.613057 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.613035    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 17:24:54.623313 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.623294    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtq92\" (UniqueName: \"kubernetes.io/projected/c1833f06-67dc-49c7-b432-7a61d45b5319-kube-api-access-mtq92\") pod \"network-check-target-9hknl\" (UID: \"c1833f06-67dc-49c7-b432-7a61d45b5319\") " pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:54.683484 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.683419    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bbbdj"]
Sep 10 17:24:54.686456 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:24:54.686431    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded270dc5_1f93_4739_b3b1_d5a1412d6819.slice/crio-cfdae0836ca4e408da295ad09f199f4f34375c2bdb336129976fc59944d5816d WatchSource:0}: Error finding container cfdae0836ca4e408da295ad09f199f4f34375c2bdb336129976fc59944d5816d: Status 404 returned error can't find the container with id cfdae0836ca4e408da295ad09f199f4f34375c2bdb336129976fc59944d5816d
Sep 10 17:24:54.712872 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.712850    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v58r5\""
Sep 10 17:24:54.721138 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.721121    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:24:54.987484 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:54.987402    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9hknl"]
Sep 10 17:24:54.989973 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:24:54.989944    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1833f06_67dc_49c7_b432_7a61d45b5319.slice/crio-8e9390780bfc6e076522091bddbaf606d868aa7e900b23e28ac21de60ca77e2a WatchSource:0}: Error finding container 8e9390780bfc6e076522091bddbaf606d868aa7e900b23e28ac21de60ca77e2a: Status 404 returned error can't find the container with id 8e9390780bfc6e076522091bddbaf606d868aa7e900b23e28ac21de60ca77e2a
Sep 10 17:24:55.211812 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:55.211777    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9hknl" event={"ID":"c1833f06-67dc-49c7-b432-7a61d45b5319","Type":"ContainerStarted","Data":"8e9390780bfc6e076522091bddbaf606d868aa7e900b23e28ac21de60ca77e2a"}
Sep 10 17:24:55.212778 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:24:55.212758    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bbbdj" event={"ID":"ed270dc5-1f93-4739-b3b1-d5a1412d6819","Type":"ContainerStarted","Data":"cfdae0836ca4e408da295ad09f199f4f34375c2bdb336129976fc59944d5816d"}
Sep 10 17:25:00.227342 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:00.227311    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9hknl" event={"ID":"c1833f06-67dc-49c7-b432-7a61d45b5319","Type":"ContainerStarted","Data":"e1de13f87f85e56ac364aa779d2dca8b1905afd0f78c1f199c2ea62f3e51018e"}
Sep 10 17:25:00.227837 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:00.227445    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:25:00.245422 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:00.245375    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9hknl" podStartSLOduration=66.074672025 podStartE2EDuration="1m11.245359149s" totalImagesPullingTime="5.170687124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:00.24452503 +0000 UTC m=+72.003052099" watchObservedRunningTime="2026-09-10 17:25:00.245359149 +0000 UTC m=+72.003886213"
Sep 10 17:25:01.231294 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:01.231255    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bbbdj" event={"ID":"ed270dc5-1f93-4739-b3b1-d5a1412d6819","Type":"ContainerStarted","Data":"662a1bc28f9908d77be28063987f78fa6000606d357318e0ec2ce786cb65e5e1"}
Sep 10 17:25:01.247970 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:01.247916    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bbbdj" podStartSLOduration=66.770375635 podStartE2EDuration="1m12.247900436s" totalImagesPullingTime="5.477524801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:01.247387039 +0000 UTC m=+73.005914109" watchObservedRunningTime="2026-09-10 17:25:01.247900436 +0000 UTC m=+73.006427505"
Sep 10 17:25:26.527225 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:26.527173    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:25:26.527225 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:26.527236    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:25:26.527696 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:26.527338    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:25:26.527696 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:26.527352    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f498f4fc-7k7bl: secret "image-registry-tls" not found
Sep 10 17:25:26.527696 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:26.527405    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls podName:47b034f8-d52d-40ca-9c28-4e2931eb61b3 nodeName:}" failed. No retries permitted until 2026-09-10 17:26:30.527392169 +0000 UTC m=+162.285919217 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls") pod "image-registry-64f498f4fc-7k7bl" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3") : secret "image-registry-tls" not found
Sep 10 17:25:26.527696 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:26.527337    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:25:26.527696 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:26.527483    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls podName:a3b1670d-12c0-4614-a86e-22c5498e1274 nodeName:}" failed. No retries permitted until 2026-09-10 17:26:30.527469527 +0000 UTC m=+162.285996576 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls") pod "dns-default-77x6z" (UID: "a3b1670d-12c0-4614-a86e-22c5498e1274") : secret "dns-default-metrics-tls" not found
Sep 10 17:25:27.706578 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.706545    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn"]
Sep 10 17:25:27.709163 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.709145    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn"
Sep 10 17:25:27.711856 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.711829    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 10 17:25:27.712931 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.712910    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:25:27.713049 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.712930    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 10 17:25:27.713049 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.712940    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 10 17:25:27.713049 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.712932    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m49q4\""
Sep 10 17:25:27.716770 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.716748    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-mwgvb"]
Sep 10 17:25:27.718934 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.718917    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:27.721999 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.721981    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 17:25:27.722085 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.721989    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 17:25:27.722281 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.722265    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bv64n\""
Sep 10 17:25:27.723034 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.723012    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:25:27.723034 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.723030    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 17:25:27.728395 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.728375    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn"]
Sep 10 17:25:27.730738 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.730716    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 17:25:27.736903 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.736884    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-mwgvb"]
Sep 10 17:25:27.834956 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.834922    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/846b39a3-9507-423f-ba80-1bbff55a4e44-serving-cert\") pod \"service-ca-operator-859f94c4b-jh5dn\" (UID: \"846b39a3-9507-423f-ba80-1bbff55a4e44\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn"
Sep 10 17:25:27.835121 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.834968    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdzrp\" (UniqueName: \"kubernetes.io/projected/846b39a3-9507-423f-ba80-1bbff55a4e44-kube-api-access-cdzrp\") pod \"service-ca-operator-859f94c4b-jh5dn\" (UID: \"846b39a3-9507-423f-ba80-1bbff55a4e44\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn"
Sep 10 17:25:27.835121 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.834998    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/89a48e5b-848f-46fa-a181-4d4b331761be-serving-cert\") pod \"console-operator-85b454874b-mwgvb\" (UID: \"89a48e5b-848f-46fa-a181-4d4b331761be\") " pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:27.835121 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.835050    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/89a48e5b-848f-46fa-a181-4d4b331761be-trusted-ca\") pod \"console-operator-85b454874b-mwgvb\" (UID: \"89a48e5b-848f-46fa-a181-4d4b331761be\") " pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:27.835121 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.835078    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89a48e5b-848f-46fa-a181-4d4b331761be-config\") pod \"console-operator-85b454874b-mwgvb\" (UID: \"89a48e5b-848f-46fa-a181-4d4b331761be\") " pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:27.835121 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.835095    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gthph\" (UniqueName: \"kubernetes.io/projected/89a48e5b-848f-46fa-a181-4d4b331761be-kube-api-access-gthph\") pod \"console-operator-85b454874b-mwgvb\" (UID: \"89a48e5b-848f-46fa-a181-4d4b331761be\") " pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:27.835121 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.835116    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/846b39a3-9507-423f-ba80-1bbff55a4e44-config\") pod \"service-ca-operator-859f94c4b-jh5dn\" (UID: \"846b39a3-9507-423f-ba80-1bbff55a4e44\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn"
Sep 10 17:25:27.936435 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.936380    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/846b39a3-9507-423f-ba80-1bbff55a4e44-serving-cert\") pod \"service-ca-operator-859f94c4b-jh5dn\" (UID: \"846b39a3-9507-423f-ba80-1bbff55a4e44\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn"
Sep 10 17:25:27.936577 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.936459    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdzrp\" (UniqueName: \"kubernetes.io/projected/846b39a3-9507-423f-ba80-1bbff55a4e44-kube-api-access-cdzrp\") pod \"service-ca-operator-859f94c4b-jh5dn\" (UID: \"846b39a3-9507-423f-ba80-1bbff55a4e44\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn"
Sep 10 17:25:27.936577 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.936518    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/89a48e5b-848f-46fa-a181-4d4b331761be-serving-cert\") pod \"console-operator-85b454874b-mwgvb\" (UID: \"89a48e5b-848f-46fa-a181-4d4b331761be\") " pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:27.936690 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.936667    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/89a48e5b-848f-46fa-a181-4d4b331761be-trusted-ca\") pod \"console-operator-85b454874b-mwgvb\" (UID: \"89a48e5b-848f-46fa-a181-4d4b331761be\") " pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:27.936737 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.936708    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89a48e5b-848f-46fa-a181-4d4b331761be-config\") pod \"console-operator-85b454874b-mwgvb\" (UID: \"89a48e5b-848f-46fa-a181-4d4b331761be\") " pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:27.936737 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.936733    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gthph\" (UniqueName: \"kubernetes.io/projected/89a48e5b-848f-46fa-a181-4d4b331761be-kube-api-access-gthph\") pod \"console-operator-85b454874b-mwgvb\" (UID: \"89a48e5b-848f-46fa-a181-4d4b331761be\") " pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:27.936861 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.936767    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/846b39a3-9507-423f-ba80-1bbff55a4e44-config\") pod \"service-ca-operator-859f94c4b-jh5dn\" (UID: \"846b39a3-9507-423f-ba80-1bbff55a4e44\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn"
Sep 10 17:25:27.937364 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.937339    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/846b39a3-9507-423f-ba80-1bbff55a4e44-config\") pod \"service-ca-operator-859f94c4b-jh5dn\" (UID: \"846b39a3-9507-423f-ba80-1bbff55a4e44\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn"
Sep 10 17:25:27.937527 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.937400    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89a48e5b-848f-46fa-a181-4d4b331761be-config\") pod \"console-operator-85b454874b-mwgvb\" (UID: \"89a48e5b-848f-46fa-a181-4d4b331761be\") " pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:27.937821 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.937798    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/89a48e5b-848f-46fa-a181-4d4b331761be-trusted-ca\") pod \"console-operator-85b454874b-mwgvb\" (UID: \"89a48e5b-848f-46fa-a181-4d4b331761be\") " pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:27.938936 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.938910    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/846b39a3-9507-423f-ba80-1bbff55a4e44-serving-cert\") pod \"service-ca-operator-859f94c4b-jh5dn\" (UID: \"846b39a3-9507-423f-ba80-1bbff55a4e44\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn"
Sep 10 17:25:27.939024 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.939006    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/89a48e5b-848f-46fa-a181-4d4b331761be-serving-cert\") pod \"console-operator-85b454874b-mwgvb\" (UID: \"89a48e5b-848f-46fa-a181-4d4b331761be\") " pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:27.947643 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.946146    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gthph\" (UniqueName: \"kubernetes.io/projected/89a48e5b-848f-46fa-a181-4d4b331761be-kube-api-access-gthph\") pod \"console-operator-85b454874b-mwgvb\" (UID: \"89a48e5b-848f-46fa-a181-4d4b331761be\") " pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:27.947643 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:27.946878    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdzrp\" (UniqueName: \"kubernetes.io/projected/846b39a3-9507-423f-ba80-1bbff55a4e44-kube-api-access-cdzrp\") pod \"service-ca-operator-859f94c4b-jh5dn\" (UID: \"846b39a3-9507-423f-ba80-1bbff55a4e44\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn"
Sep 10 17:25:28.017890 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:28.017797    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn"
Sep 10 17:25:28.027485 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:28.027457    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:28.287278 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:28.287202    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn"]
Sep 10 17:25:28.290667 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:25:28.290629    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846b39a3_9507_423f_ba80_1bbff55a4e44.slice/crio-c6829b50901332f8db436cffc079f8c7d54c56624e8b129a5f27642555a877e7 WatchSource:0}: Error finding container c6829b50901332f8db436cffc079f8c7d54c56624e8b129a5f27642555a877e7: Status 404 returned error can't find the container with id c6829b50901332f8db436cffc079f8c7d54c56624e8b129a5f27642555a877e7
Sep 10 17:25:28.297091 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:28.297062    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn" event={"ID":"846b39a3-9507-423f-ba80-1bbff55a4e44","Type":"ContainerStarted","Data":"c6829b50901332f8db436cffc079f8c7d54c56624e8b129a5f27642555a877e7"}
Sep 10 17:25:28.307323 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:28.307299    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-mwgvb"]
Sep 10 17:25:28.310069 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:25:28.310043    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a48e5b_848f_46fa_a181_4d4b331761be.slice/crio-7119a9d09903c24cd7212f942bac0ef1c4e1a30e03582c55732c7cb337759960 WatchSource:0}: Error finding container 7119a9d09903c24cd7212f942bac0ef1c4e1a30e03582c55732c7cb337759960: Status 404 returned error can't find the container with id 7119a9d09903c24cd7212f942bac0ef1c4e1a30e03582c55732c7cb337759960
Sep 10 17:25:29.301112 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:29.301080    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-mwgvb" event={"ID":"89a48e5b-848f-46fa-a181-4d4b331761be","Type":"ContainerStarted","Data":"7119a9d09903c24cd7212f942bac0ef1c4e1a30e03582c55732c7cb337759960"}
Sep 10 17:25:31.233900 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.233875    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9hknl"
Sep 10 17:25:31.309470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.309381    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-mwgvb" event={"ID":"89a48e5b-848f-46fa-a181-4d4b331761be","Type":"ContainerStarted","Data":"e92a6c27693192420a3c96595447d49ce8ca3c297907b63e7021d8657858e7a7"}
Sep 10 17:25:31.309720 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.309654    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:31.310987 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.310938    2552 patch_prober.go:28] interesting pod/console-operator-85b454874b-mwgvb container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body=
Sep 10 17:25:31.311106 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.310994    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-85b454874b-mwgvb" podUID="89a48e5b-848f-46fa-a181-4d4b331761be" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused"
Sep 10 17:25:31.311463 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.311415    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn" event={"ID":"846b39a3-9507-423f-ba80-1bbff55a4e44","Type":"ContainerStarted","Data":"2c4f2c19ecda3cc86590d95b44a8b2f766c37788c36a78d30761c28d1c0bac51"}
Sep 10 17:25:31.326968 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.326723    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-mwgvb" podStartSLOduration=1.455139255 podStartE2EDuration="4.326710048s" totalImagesPullingTime="2.871570793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:31.32640123 +0000 UTC m=+103.084928299" watchObservedRunningTime="2026-09-10 17:25:31.326710048 +0000 UTC m=+103.085237117"
Sep 10 17:25:31.342695 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.342634    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn" podStartSLOduration=1.452565029 podStartE2EDuration="4.34261728s" totalImagesPullingTime="2.890052251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:31.341966326 +0000 UTC m=+103.100493388" watchObservedRunningTime="2026-09-10 17:25:31.34261728 +0000 UTC m=+103.101144349"
Sep 10 17:25:31.896810 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.896759    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-glx7s"]
Sep 10 17:25:31.917669 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.917639    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:31.919783 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.919746    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-glx7s"]
Sep 10 17:25:31.924220 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.924021    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 17:25:31.924766 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.924059    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 17:25:31.924766 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.924218    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-82xn6\""
Sep 10 17:25:31.925203 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.925183    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 17:25:31.925417 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.925399    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 17:25:31.967454 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.967417    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wx27\" (UniqueName: \"kubernetes.io/projected/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-api-access-4wx27\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:31.967679 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.967484    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:31.967679 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.967528    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:31.967679 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.967598    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d163bbc-7b58-4c6d-9ced-45db33946115-data-volume\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:31.967679 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:31.967629    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6d163bbc-7b58-4c6d-9ced-45db33946115-crio-socket\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:32.068488 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:32.068445    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6d163bbc-7b58-4c6d-9ced-45db33946115-crio-socket\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:32.068488 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:32.068500    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wx27\" (UniqueName: \"kubernetes.io/projected/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-api-access-4wx27\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:32.068769 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:32.068583    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6d163bbc-7b58-4c6d-9ced-45db33946115-crio-socket\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:32.068769 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:32.068632    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:32.068769 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:32.068674    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:32.068769 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:32.068687    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:25:32.068769 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:32.068729    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d163bbc-7b58-4c6d-9ced-45db33946115-data-volume\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:32.068769 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:32.068753    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm podName:6d163bbc-7b58-4c6d-9ced-45db33946115 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:32.568737866 +0000 UTC m=+104.327264930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm") pod "insights-runtime-extractor-glx7s" (UID: "6d163bbc-7b58-4c6d-9ced-45db33946115") : configmap "kube-rbac-proxy" not found
Sep 10 17:25:32.069005 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:32.068785    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:32.069005 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:32.068834    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls podName:6d163bbc-7b58-4c6d-9ced-45db33946115 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:32.568819075 +0000 UTC m=+104.327346138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls") pod "insights-runtime-extractor-glx7s" (UID: "6d163bbc-7b58-4c6d-9ced-45db33946115") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:32.069080 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:32.069005    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d163bbc-7b58-4c6d-9ced-45db33946115-data-volume\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:32.092225 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:32.092192    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wx27\" (UniqueName: \"kubernetes.io/projected/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-api-access-4wx27\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:32.315958 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:32.315884    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/0.log"
Sep 10 17:25:32.315958 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:32.315930    2552 generic.go:356] "Generic (PLEG): container finished" podID="89a48e5b-848f-46fa-a181-4d4b331761be" containerID="e92a6c27693192420a3c96595447d49ce8ca3c297907b63e7021d8657858e7a7" exitCode=255
Sep 10 17:25:32.316378 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:32.316023    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-mwgvb" event={"ID":"89a48e5b-848f-46fa-a181-4d4b331761be","Type":"ContainerDied","Data":"e92a6c27693192420a3c96595447d49ce8ca3c297907b63e7021d8657858e7a7"}
Sep 10 17:25:32.316378 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:32.316294    2552 scope.go:117] "RemoveContainer" containerID="e92a6c27693192420a3c96595447d49ce8ca3c297907b63e7021d8657858e7a7"
Sep 10 17:25:32.572777 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:32.572678    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:32.572777 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:32.572725    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:32.573001 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:32.572799    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:25:32.573001 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:32.572814    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:32.573001 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:32.572870    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm podName:6d163bbc-7b58-4c6d-9ced-45db33946115 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:33.572853048 +0000 UTC m=+105.331380096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm") pod "insights-runtime-extractor-glx7s" (UID: "6d163bbc-7b58-4c6d-9ced-45db33946115") : configmap "kube-rbac-proxy" not found
Sep 10 17:25:32.573001 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:32.572883    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls podName:6d163bbc-7b58-4c6d-9ced-45db33946115 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:33.572877454 +0000 UTC m=+105.331404502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls") pod "insights-runtime-extractor-glx7s" (UID: "6d163bbc-7b58-4c6d-9ced-45db33946115") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:33.321003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:33.320971    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:25:33.321372 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:33.321356    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/0.log"
Sep 10 17:25:33.321419 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:33.321390    2552 generic.go:356] "Generic (PLEG): container finished" podID="89a48e5b-848f-46fa-a181-4d4b331761be" containerID="050f0c548f4c92f3634ab4637c9a92217e6dc04707065f304735186a486dc118" exitCode=255
Sep 10 17:25:33.321453 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:33.321422    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-mwgvb" event={"ID":"89a48e5b-848f-46fa-a181-4d4b331761be","Type":"ContainerDied","Data":"050f0c548f4c92f3634ab4637c9a92217e6dc04707065f304735186a486dc118"}
Sep 10 17:25:33.321522 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:33.321507    2552 scope.go:117] "RemoveContainer" containerID="e92a6c27693192420a3c96595447d49ce8ca3c297907b63e7021d8657858e7a7"
Sep 10 17:25:33.321754 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:33.321735    2552 scope.go:117] "RemoveContainer" containerID="050f0c548f4c92f3634ab4637c9a92217e6dc04707065f304735186a486dc118"
Sep 10 17:25:33.321904 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:33.321888    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-mwgvb_openshift-console-operator(89a48e5b-848f-46fa-a181-4d4b331761be)\"" pod="openshift-console-operator/console-operator-85b454874b-mwgvb" podUID="89a48e5b-848f-46fa-a181-4d4b331761be"
Sep 10 17:25:33.580371 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:33.580272    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:33.580371 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:33.580315    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:33.580630 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:33.580403    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:25:33.580630 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:33.580428    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:33.580630 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:33.580484    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm podName:6d163bbc-7b58-4c6d-9ced-45db33946115 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:35.580462534 +0000 UTC m=+107.338989586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm") pod "insights-runtime-extractor-glx7s" (UID: "6d163bbc-7b58-4c6d-9ced-45db33946115") : configmap "kube-rbac-proxy" not found
Sep 10 17:25:33.580630 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:33.580521    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls podName:6d163bbc-7b58-4c6d-9ced-45db33946115 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:35.580514373 +0000 UTC m=+107.339041421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls") pod "insights-runtime-extractor-glx7s" (UID: "6d163bbc-7b58-4c6d-9ced-45db33946115") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:34.053884 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:34.053856    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f9tg8_81246f30-ad68-472c-aa8b-72d14634f501/dns-node-resolver/0.log"
Sep 10 17:25:34.325997 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:34.325916    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:25:34.326436 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:34.326283    2552 scope.go:117] "RemoveContainer" containerID="050f0c548f4c92f3634ab4637c9a92217e6dc04707065f304735186a486dc118"
Sep 10 17:25:34.326519 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:34.326463    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-mwgvb_openshift-console-operator(89a48e5b-848f-46fa-a181-4d4b331761be)\"" pod="openshift-console-operator/console-operator-85b454874b-mwgvb" podUID="89a48e5b-848f-46fa-a181-4d4b331761be"
Sep 10 17:25:35.051357 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.051325    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wwbzr_7067d92a-5363-42fe-a8f0-7c765931c532/node-ca/0.log"
Sep 10 17:25:35.416134 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.416097    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-fd624"]
Sep 10 17:25:35.418725 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.418705    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-fd624"
Sep 10 17:25:35.421608 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.421590    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 17:25:35.422903 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.422886    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 17:25:35.422987 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.422888    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hfmwf\""
Sep 10 17:25:35.422987 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.422915    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 17:25:35.422987 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.422887    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 17:25:35.429228 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.429206    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-fd624"]
Sep 10 17:25:35.450866 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.450821    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-55x6q_fff24ddc-90d7-4565-8f85-27c6f028cca4/serve-healthcheck-canary/0.log"
Sep 10 17:25:35.496043 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.495999    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5-signing-key\") pod \"service-ca-75d99c484-fd624\" (UID: \"3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5\") " pod="openshift-service-ca/service-ca-75d99c484-fd624"
Sep 10 17:25:35.496185 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.496146    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5-signing-cabundle\") pod \"service-ca-75d99c484-fd624\" (UID: \"3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5\") " pod="openshift-service-ca/service-ca-75d99c484-fd624"
Sep 10 17:25:35.496236 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.496196    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngg7w\" (UniqueName: \"kubernetes.io/projected/3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5-kube-api-access-ngg7w\") pod \"service-ca-75d99c484-fd624\" (UID: \"3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5\") " pod="openshift-service-ca/service-ca-75d99c484-fd624"
Sep 10 17:25:35.597129 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.597087    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5-signing-cabundle\") pod \"service-ca-75d99c484-fd624\" (UID: \"3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5\") " pod="openshift-service-ca/service-ca-75d99c484-fd624"
Sep 10 17:25:35.597129 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.597130    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:35.597369 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.597166    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngg7w\" (UniqueName: \"kubernetes.io/projected/3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5-kube-api-access-ngg7w\") pod \"service-ca-75d99c484-fd624\" (UID: \"3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5\") " pod="openshift-service-ca/service-ca-75d99c484-fd624"
Sep 10 17:25:35.597369 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.597206    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:35.597369 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:35.597225    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:25:35.597369 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.597236    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5-signing-key\") pod \"service-ca-75d99c484-fd624\" (UID: \"3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5\") " pod="openshift-service-ca/service-ca-75d99c484-fd624"
Sep 10 17:25:35.597369 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:35.597294    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm podName:6d163bbc-7b58-4c6d-9ced-45db33946115 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:39.597277646 +0000 UTC m=+111.355804695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm") pod "insights-runtime-extractor-glx7s" (UID: "6d163bbc-7b58-4c6d-9ced-45db33946115") : configmap "kube-rbac-proxy" not found
Sep 10 17:25:35.597369 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:35.597346    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:35.597690 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:35.597411    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls podName:6d163bbc-7b58-4c6d-9ced-45db33946115 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:39.597395479 +0000 UTC m=+111.355922526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls") pod "insights-runtime-extractor-glx7s" (UID: "6d163bbc-7b58-4c6d-9ced-45db33946115") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:35.597795 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.597774    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5-signing-cabundle\") pod \"service-ca-75d99c484-fd624\" (UID: \"3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5\") " pod="openshift-service-ca/service-ca-75d99c484-fd624"
Sep 10 17:25:35.599564 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.599543    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5-signing-key\") pod \"service-ca-75d99c484-fd624\" (UID: \"3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5\") " pod="openshift-service-ca/service-ca-75d99c484-fd624"
Sep 10 17:25:35.607705 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.607685    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngg7w\" (UniqueName: \"kubernetes.io/projected/3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5-kube-api-access-ngg7w\") pod \"service-ca-75d99c484-fd624\" (UID: \"3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5\") " pod="openshift-service-ca/service-ca-75d99c484-fd624"
Sep 10 17:25:35.727663 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.727564    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-fd624"
Sep 10 17:25:35.998538 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:35.998435    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-fd624"]
Sep 10 17:25:36.002224 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:25:36.002193    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d88b6ef_d03d_46ca_8f93_ffbcd64b7cc5.slice/crio-3770cca1e77e290a73cb95ed6e7f574f2fa819bac654f0d6a6e8a98060b8d072 WatchSource:0}: Error finding container 3770cca1e77e290a73cb95ed6e7f574f2fa819bac654f0d6a6e8a98060b8d072: Status 404 returned error can't find the container with id 3770cca1e77e290a73cb95ed6e7f574f2fa819bac654f0d6a6e8a98060b8d072
Sep 10 17:25:36.333840 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:36.333751    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-fd624" event={"ID":"3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5","Type":"ContainerStarted","Data":"68e4d0f42c60885906515fa520a6637ed10a6b5234d69f372863cc73cb1a32ef"}
Sep 10 17:25:36.333840 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:36.333812    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-fd624" event={"ID":"3d88b6ef-d03d-46ca-8f93-ffbcd64b7cc5","Type":"ContainerStarted","Data":"3770cca1e77e290a73cb95ed6e7f574f2fa819bac654f0d6a6e8a98060b8d072"}
Sep 10 17:25:36.354788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:36.354740    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-fd624" podStartSLOduration=1.354725057 podStartE2EDuration="1.354725057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:36.353234065 +0000 UTC m=+108.111761136" watchObservedRunningTime="2026-09-10 17:25:36.354725057 +0000 UTC m=+108.113252187"
Sep 10 17:25:38.028072 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:38.028031    2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:38.028481 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:38.028461    2552 scope.go:117] "RemoveContainer" containerID="050f0c548f4c92f3634ab4637c9a92217e6dc04707065f304735186a486dc118"
Sep 10 17:25:38.028652 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:38.028633    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-mwgvb_openshift-console-operator(89a48e5b-848f-46fa-a181-4d4b331761be)\"" pod="openshift-console-operator/console-operator-85b454874b-mwgvb" podUID="89a48e5b-848f-46fa-a181-4d4b331761be"
Sep 10 17:25:39.631709 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:39.631667    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:39.632096 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:39.631720    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:39.632096 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:39.631819    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:25:39.632096 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:39.631826    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:39.632096 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:39.631886    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm podName:6d163bbc-7b58-4c6d-9ced-45db33946115 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:47.631870151 +0000 UTC m=+119.390397199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm") pod "insights-runtime-extractor-glx7s" (UID: "6d163bbc-7b58-4c6d-9ced-45db33946115") : configmap "kube-rbac-proxy" not found
Sep 10 17:25:39.632096 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:39.631937    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls podName:6d163bbc-7b58-4c6d-9ced-45db33946115 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:47.631916646 +0000 UTC m=+119.390443699 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls") pod "insights-runtime-extractor-glx7s" (UID: "6d163bbc-7b58-4c6d-9ced-45db33946115") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:41.310595 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:41.310561    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:41.310962 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:41.310939    2552 scope.go:117] "RemoveContainer" containerID="050f0c548f4c92f3634ab4637c9a92217e6dc04707065f304735186a486dc118"
Sep 10 17:25:41.311111 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:25:41.311091    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-mwgvb_openshift-console-operator(89a48e5b-848f-46fa-a181-4d4b331761be)\"" pod="openshift-console-operator/console-operator-85b454874b-mwgvb" podUID="89a48e5b-848f-46fa-a181-4d4b331761be"
Sep 10 17:25:47.703160 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:47.703116    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:47.703549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:47.703173    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:47.703674 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:47.703653    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6d163bbc-7b58-4c6d-9ced-45db33946115-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:47.705431 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:47.705413    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d163bbc-7b58-4c6d-9ced-45db33946115-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glx7s\" (UID: \"6d163bbc-7b58-4c6d-9ced-45db33946115\") " pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:47.827316 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:47.827282    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-glx7s"
Sep 10 17:25:48.095662 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:48.095640    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-glx7s"]
Sep 10 17:25:48.098187 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:25:48.098162    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d163bbc_7b58_4c6d_9ced_45db33946115.slice/crio-acae3726a842fa71e69f4c00da3d56c2a205c34735f31cdb988b74baba42b50f WatchSource:0}: Error finding container acae3726a842fa71e69f4c00da3d56c2a205c34735f31cdb988b74baba42b50f: Status 404 returned error can't find the container with id acae3726a842fa71e69f4c00da3d56c2a205c34735f31cdb988b74baba42b50f
Sep 10 17:25:48.368526 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:48.368425    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glx7s" event={"ID":"6d163bbc-7b58-4c6d-9ced-45db33946115","Type":"ContainerStarted","Data":"9fe043bd2a4a7c2a5105c241d35bdbf30a4dc82604f66fd7d59073814884719f"}
Sep 10 17:25:48.368526 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:48.368487    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glx7s" event={"ID":"6d163bbc-7b58-4c6d-9ced-45db33946115","Type":"ContainerStarted","Data":"acae3726a842fa71e69f4c00da3d56c2a205c34735f31cdb988b74baba42b50f"}
Sep 10 17:25:49.373271 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:49.373240    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glx7s" event={"ID":"6d163bbc-7b58-4c6d-9ced-45db33946115","Type":"ContainerStarted","Data":"0fac0bc967b4513cd977ea2e08c7fc6c00653953f457cdf69d51363a2f43497a"}
Sep 10 17:25:51.383256 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:51.383208    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glx7s" event={"ID":"6d163bbc-7b58-4c6d-9ced-45db33946115","Type":"ContainerStarted","Data":"e9ddac93d2089e1ae10ea4cd3cfa656358a8d1fd19427a1dfd6495e155720482"}
Sep 10 17:25:51.402225 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:51.402180    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-glx7s" podStartSLOduration=18.237838891 podStartE2EDuration="20.402167608s" totalImagesPullingTime="2.164328717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:25:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:51.401912895 +0000 UTC m=+123.160439965" watchObservedRunningTime="2026-09-10 17:25:51.402167608 +0000 UTC m=+123.160694678"
Sep 10 17:25:54.797703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:54.797672    2552 scope.go:117] "RemoveContainer" containerID="050f0c548f4c92f3634ab4637c9a92217e6dc04707065f304735186a486dc118"
Sep 10 17:25:55.395093 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:55.395064    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:25:55.395261 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:55.395117    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-mwgvb" event={"ID":"89a48e5b-848f-46fa-a181-4d4b331761be","Type":"ContainerStarted","Data":"aee41dd55069d058a20048d6dbd0032770bde39c491907863d58e091a7acd775"}
Sep 10 17:25:55.395394 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:55.395373    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:55.399821 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:55.399797    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-mwgvb"
Sep 10 17:25:57.237558 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.237521    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"]
Sep 10 17:25:57.239728 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.239705    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-clnst"]
Sep 10 17:25:57.239874 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.239834    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"
Sep 10 17:25:57.241602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.241582    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-clnst"
Sep 10 17:25:57.244140 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.244118    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 17:25:57.244270 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.244139    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 17:25:57.244270 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.244250    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 10 17:25:57.244959 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.244931    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2znkh\""
Sep 10 17:25:57.244959 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.244946    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 17:25:57.245140 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.244963    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 10 17:25:57.245140 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.244948    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 17:25:57.257863 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.257842    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"]
Sep 10 17:25:57.258731 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.258709    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-clnst"]
Sep 10 17:25:57.341118 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.341084    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-ccl54"]
Sep 10 17:25:57.343021 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.342996    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-ccl54"
Sep 10 17:25:57.345520 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.345483    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 17:25:57.345653 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.345567    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-252bf\""
Sep 10 17:25:57.345780 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.345767    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 17:25:57.358175 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.358150    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-ccl54"]
Sep 10 17:25:57.377173 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.377142    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29057183-7c39-4679-80cd-aa4d4c2deebe-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-clnst\" (UID: \"29057183-7c39-4679-80cd-aa4d4c2deebe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-clnst"
Sep 10 17:25:57.377328 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.377180    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq8ft\" (UniqueName: \"kubernetes.io/projected/7114de69-9a4a-4c6a-abb4-4753f1a7a407-kube-api-access-pq8ft\") pod \"klusterlet-addon-workmgr-c9f8cbbb8-kfqh8\" (UID: \"7114de69-9a4a-4c6a-abb4-4753f1a7a407\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"
Sep 10 17:25:57.377328 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.377209    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/29057183-7c39-4679-80cd-aa4d4c2deebe-nginx-conf\") pod \"networking-console-plugin-79977f9844-clnst\" (UID: \"29057183-7c39-4679-80cd-aa4d4c2deebe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-clnst"
Sep 10 17:25:57.377328 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.377257    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7114de69-9a4a-4c6a-abb4-4753f1a7a407-klusterlet-config\") pod \"klusterlet-addon-workmgr-c9f8cbbb8-kfqh8\" (UID: \"7114de69-9a4a-4c6a-abb4-4753f1a7a407\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"
Sep 10 17:25:57.377328 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.377319    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7114de69-9a4a-4c6a-abb4-4753f1a7a407-tmp\") pod \"klusterlet-addon-workmgr-c9f8cbbb8-kfqh8\" (UID: \"7114de69-9a4a-4c6a-abb4-4753f1a7a407\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"
Sep 10 17:25:57.478160 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.478127    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq8ft\" (UniqueName: \"kubernetes.io/projected/7114de69-9a4a-4c6a-abb4-4753f1a7a407-kube-api-access-pq8ft\") pod \"klusterlet-addon-workmgr-c9f8cbbb8-kfqh8\" (UID: \"7114de69-9a4a-4c6a-abb4-4753f1a7a407\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"
Sep 10 17:25:57.478354 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.478171    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/29057183-7c39-4679-80cd-aa4d4c2deebe-nginx-conf\") pod \"networking-console-plugin-79977f9844-clnst\" (UID: \"29057183-7c39-4679-80cd-aa4d4c2deebe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-clnst"
Sep 10 17:25:57.478354 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.478188    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7114de69-9a4a-4c6a-abb4-4753f1a7a407-klusterlet-config\") pod \"klusterlet-addon-workmgr-c9f8cbbb8-kfqh8\" (UID: \"7114de69-9a4a-4c6a-abb4-4753f1a7a407\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"
Sep 10 17:25:57.478354 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.478222    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7114de69-9a4a-4c6a-abb4-4753f1a7a407-tmp\") pod \"klusterlet-addon-workmgr-c9f8cbbb8-kfqh8\" (UID: \"7114de69-9a4a-4c6a-abb4-4753f1a7a407\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"
Sep 10 17:25:57.478354 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.478268    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29057183-7c39-4679-80cd-aa4d4c2deebe-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-clnst\" (UID: \"29057183-7c39-4679-80cd-aa4d4c2deebe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-clnst"
Sep 10 17:25:57.478354 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.478289    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7fsq\" (UniqueName: \"kubernetes.io/projected/59a60e4b-21b8-4ede-b64c-8164dc12742c-kube-api-access-v7fsq\") pod \"downloads-85d98c998f-ccl54\" (UID: \"59a60e4b-21b8-4ede-b64c-8164dc12742c\") " pod="openshift-console/downloads-85d98c998f-ccl54"
Sep 10 17:25:57.478672 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.478646    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7114de69-9a4a-4c6a-abb4-4753f1a7a407-tmp\") pod \"klusterlet-addon-workmgr-c9f8cbbb8-kfqh8\" (UID: \"7114de69-9a4a-4c6a-abb4-4753f1a7a407\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"
Sep 10 17:25:57.478893 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.478870    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/29057183-7c39-4679-80cd-aa4d4c2deebe-nginx-conf\") pod \"networking-console-plugin-79977f9844-clnst\" (UID: \"29057183-7c39-4679-80cd-aa4d4c2deebe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-clnst"
Sep 10 17:25:57.480720 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.480703    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29057183-7c39-4679-80cd-aa4d4c2deebe-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-clnst\" (UID: \"29057183-7c39-4679-80cd-aa4d4c2deebe\") " pod="openshift-network-console/networking-console-plugin-79977f9844-clnst"
Sep 10 17:25:57.480923 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.480905    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7114de69-9a4a-4c6a-abb4-4753f1a7a407-klusterlet-config\") pod \"klusterlet-addon-workmgr-c9f8cbbb8-kfqh8\" (UID: \"7114de69-9a4a-4c6a-abb4-4753f1a7a407\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"
Sep 10 17:25:57.491242 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.491178    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq8ft\" (UniqueName: \"kubernetes.io/projected/7114de69-9a4a-4c6a-abb4-4753f1a7a407-kube-api-access-pq8ft\") pod \"klusterlet-addon-workmgr-c9f8cbbb8-kfqh8\" (UID: \"7114de69-9a4a-4c6a-abb4-4753f1a7a407\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"
Sep 10 17:25:57.552934 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.552884    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"
Sep 10 17:25:57.559118 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.559093    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-clnst"
Sep 10 17:25:57.579443 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.579403    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7fsq\" (UniqueName: \"kubernetes.io/projected/59a60e4b-21b8-4ede-b64c-8164dc12742c-kube-api-access-v7fsq\") pod \"downloads-85d98c998f-ccl54\" (UID: \"59a60e4b-21b8-4ede-b64c-8164dc12742c\") " pod="openshift-console/downloads-85d98c998f-ccl54"
Sep 10 17:25:57.590791 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.590758    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7fsq\" (UniqueName: \"kubernetes.io/projected/59a60e4b-21b8-4ede-b64c-8164dc12742c-kube-api-access-v7fsq\") pod \"downloads-85d98c998f-ccl54\" (UID: \"59a60e4b-21b8-4ede-b64c-8164dc12742c\") " pod="openshift-console/downloads-85d98c998f-ccl54"
Sep 10 17:25:57.653215 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.653180    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-ccl54"
Sep 10 17:25:57.827148 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.827073    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"]
Sep 10 17:25:57.830138 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:25:57.830106    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7114de69_9a4a_4c6a_abb4_4753f1a7a407.slice/crio-6cf32b6e24285d447e37c4a646c0a4fc9b0b9aa76d4bfa3b479b56172142242d WatchSource:0}: Error finding container 6cf32b6e24285d447e37c4a646c0a4fc9b0b9aa76d4bfa3b479b56172142242d: Status 404 returned error can't find the container with id 6cf32b6e24285d447e37c4a646c0a4fc9b0b9aa76d4bfa3b479b56172142242d
Sep 10 17:25:57.831007 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.830977    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-clnst"]
Sep 10 17:25:57.833351 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:25:57.833328    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29057183_7c39_4679_80cd_aa4d4c2deebe.slice/crio-a10f4308585757cd7258ef439e01706ec877c82a5c91afd5272a0f9d4c14fccf WatchSource:0}: Error finding container a10f4308585757cd7258ef439e01706ec877c82a5c91afd5272a0f9d4c14fccf: Status 404 returned error can't find the container with id a10f4308585757cd7258ef439e01706ec877c82a5c91afd5272a0f9d4c14fccf
Sep 10 17:25:57.925885 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:57.925855    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-ccl54"]
Sep 10 17:25:57.928589 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:25:57.928559    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a60e4b_21b8_4ede_b64c_8164dc12742c.slice/crio-7d31594a0d4d568e311c3f3346a25c7d30d39f5485fe893399c1e5e03f386696 WatchSource:0}: Error finding container 7d31594a0d4d568e311c3f3346a25c7d30d39f5485fe893399c1e5e03f386696: Status 404 returned error can't find the container with id 7d31594a0d4d568e311c3f3346a25c7d30d39f5485fe893399c1e5e03f386696
Sep 10 17:25:58.407379 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:58.407345    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-clnst" event={"ID":"29057183-7c39-4679-80cd-aa4d4c2deebe","Type":"ContainerStarted","Data":"a10f4308585757cd7258ef439e01706ec877c82a5c91afd5272a0f9d4c14fccf"}
Sep 10 17:25:58.408364 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:58.408340    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8" event={"ID":"7114de69-9a4a-4c6a-abb4-4753f1a7a407","Type":"ContainerStarted","Data":"6cf32b6e24285d447e37c4a646c0a4fc9b0b9aa76d4bfa3b479b56172142242d"}
Sep 10 17:25:58.409359 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:58.409342    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-ccl54" event={"ID":"59a60e4b-21b8-4ede-b64c-8164dc12742c","Type":"ContainerStarted","Data":"7d31594a0d4d568e311c3f3346a25c7d30d39f5485fe893399c1e5e03f386696"}
Sep 10 17:25:58.487970 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:58.487933    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs\") pod \"network-metrics-daemon-jk4wd\" (UID: \"3e821fe8-fb34-458c-b7d0-521a7ad7a31e\") " pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:25:58.490766 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:58.490734    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e821fe8-fb34-458c-b7d0-521a7ad7a31e-metrics-certs\") pod \"network-metrics-daemon-jk4wd\" (UID: \"3e821fe8-fb34-458c-b7d0-521a7ad7a31e\") " pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:25:58.627072 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:58.627036    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qjn9k\""
Sep 10 17:25:58.635428 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:58.635397    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jk4wd"
Sep 10 17:25:58.924313 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:58.924278    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jk4wd"]
Sep 10 17:25:59.263171 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:25:59.263132    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e821fe8_fb34_458c_b7d0_521a7ad7a31e.slice/crio-46de07314b657b229accb8e8e1aedba9983981d1dde9da497186fd0c6e5d22b3 WatchSource:0}: Error finding container 46de07314b657b229accb8e8e1aedba9983981d1dde9da497186fd0c6e5d22b3: Status 404 returned error can't find the container with id 46de07314b657b229accb8e8e1aedba9983981d1dde9da497186fd0c6e5d22b3
Sep 10 17:25:59.417158 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:59.417120    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-clnst" event={"ID":"29057183-7c39-4679-80cd-aa4d4c2deebe","Type":"ContainerStarted","Data":"d34a70dc7841c3eee062f1d4df7a1edc0e3d76cf178c775e5cd9b047ecd49533"}
Sep 10 17:25:59.419002 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:59.418974    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jk4wd" event={"ID":"3e821fe8-fb34-458c-b7d0-521a7ad7a31e","Type":"ContainerStarted","Data":"46de07314b657b229accb8e8e1aedba9983981d1dde9da497186fd0c6e5d22b3"}
Sep 10 17:25:59.433136 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:25:59.433086    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-clnst" podStartSLOduration=0.968136419 podStartE2EDuration="2.433067857s" totalImagesPullingTime="1.464931438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:59.432480519 +0000 UTC m=+131.191007590" watchObservedRunningTime="2026-09-10 17:25:59.433067857 +0000 UTC m=+131.191594929"
Sep 10 17:26:03.112885 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.112848    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-n9bcl"]
Sep 10 17:26:03.115119 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.115098    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-n9bcl"
Sep 10 17:26:03.117754 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.117728    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 10 17:26:03.117879 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.117772    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tcl99\""
Sep 10 17:26:03.125744 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.125283    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-n9bcl"]
Sep 10 17:26:03.125744 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.125649    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fb049c3e-b1ef-43b0-a466-18cd3d49575c-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-n9bcl\" (UID: \"fb049c3e-b1ef-43b0-a466-18cd3d49575c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-n9bcl"
Sep 10 17:26:03.226168 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.226129    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fb049c3e-b1ef-43b0-a466-18cd3d49575c-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-n9bcl\" (UID: \"fb049c3e-b1ef-43b0-a466-18cd3d49575c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-n9bcl"
Sep 10 17:26:03.226400 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:26:03.226303    2552 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 10 17:26:03.226400 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:26:03.226386    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb049c3e-b1ef-43b0-a466-18cd3d49575c-tls-certificates podName:fb049c3e-b1ef-43b0-a466-18cd3d49575c nodeName:}" failed. No retries permitted until 2026-09-10 17:26:03.726365528 +0000 UTC m=+135.484892594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/fb049c3e-b1ef-43b0-a466-18cd3d49575c-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-n9bcl" (UID: "fb049c3e-b1ef-43b0-a466-18cd3d49575c") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 10 17:26:03.437341 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.437242    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8" event={"ID":"7114de69-9a4a-4c6a-abb4-4753f1a7a407","Type":"ContainerStarted","Data":"fe82638f23e079bd640d44c699ee3cda8f5da11a543e8631661cf7a95fe61f86"}
Sep 10 17:26:03.437530 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.437447    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"
Sep 10 17:26:03.439543 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.439524    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8"
Sep 10 17:26:03.440063 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.440041    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jk4wd" event={"ID":"3e821fe8-fb34-458c-b7d0-521a7ad7a31e","Type":"ContainerStarted","Data":"163662bdb39bd91916dbc817115d1dc7454edba5c754b53a1e369498e8405448"}
Sep 10 17:26:03.440171 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.440083    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jk4wd" event={"ID":"3e821fe8-fb34-458c-b7d0-521a7ad7a31e","Type":"ContainerStarted","Data":"d9c29e5025898464a4cfef2de814dd2a69e14b3d7136d73c70db33be9d7b9b73"}
Sep 10 17:26:03.461982 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.461932    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c9f8cbbb8-kfqh8" podStartSLOduration=1.892032631 podStartE2EDuration="6.461917325s" totalImagesPullingTime="4.569884694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:03.461355127 +0000 UTC m=+135.219882196" watchObservedRunningTime="2026-09-10 17:26:03.461917325 +0000 UTC m=+135.220444394"
Sep 10 17:26:03.506022 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.505953    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jk4wd" podStartSLOduration=131.391314013 podStartE2EDuration="2m14.505928704s" totalImagesPullingTime="3.114614691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:03.50533826 +0000 UTC m=+135.263865343" watchObservedRunningTime="2026-09-10 17:26:03.505928704 +0000 UTC m=+135.264455775"
Sep 10 17:26:03.731484 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.731382    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fb049c3e-b1ef-43b0-a466-18cd3d49575c-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-n9bcl\" (UID: \"fb049c3e-b1ef-43b0-a466-18cd3d49575c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-n9bcl"
Sep 10 17:26:03.733843 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:03.733815    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fb049c3e-b1ef-43b0-a466-18cd3d49575c-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-n9bcl\" (UID: \"fb049c3e-b1ef-43b0-a466-18cd3d49575c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-n9bcl"
Sep 10 17:26:04.027483 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:04.027409    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-n9bcl"
Sep 10 17:26:04.301506 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:04.301417    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-n9bcl"]
Sep 10 17:26:04.304726 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:26:04.304701    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb049c3e_b1ef_43b0_a466_18cd3d49575c.slice/crio-664801b79f01df27c4edb132d099f9c7bf2baba0334c6efdde21c6f80403c46e WatchSource:0}: Error finding container 664801b79f01df27c4edb132d099f9c7bf2baba0334c6efdde21c6f80403c46e: Status 404 returned error can't find the container with id 664801b79f01df27c4edb132d099f9c7bf2baba0334c6efdde21c6f80403c46e
Sep 10 17:26:04.445582 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:04.445537    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-n9bcl" event={"ID":"fb049c3e-b1ef-43b0-a466-18cd3d49575c","Type":"ContainerStarted","Data":"664801b79f01df27c4edb132d099f9c7bf2baba0334c6efdde21c6f80403c46e"}
Sep 10 17:26:06.454902 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:06.454863    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-n9bcl" event={"ID":"fb049c3e-b1ef-43b0-a466-18cd3d49575c","Type":"ContainerStarted","Data":"06f22f704f6908374317a04d6e407a62dbe255f9bdf3bbd64014fb03e7672cf6"}
Sep 10 17:26:06.455363 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:06.455008    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-n9bcl"
Sep 10 17:26:06.460476 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:06.460425    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-n9bcl"
Sep 10 17:26:06.472004 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:06.471958    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-n9bcl" podStartSLOduration=2.300252408 podStartE2EDuration="3.471942895s" totalImagesPullingTime="1.171690487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:06.470581699 +0000 UTC m=+138.229108769" watchObservedRunningTime="2026-09-10 17:26:06.471942895 +0000 UTC m=+138.230469965"
Sep 10 17:26:07.614911 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.614877    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57db55b47c-9z44z"]
Sep 10 17:26:07.617166 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.617144    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.621602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.621573    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 17:26:07.621602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.621600    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 17:26:07.621798 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.621600    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 17:26:07.621798 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.621663    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7pmmj\""
Sep 10 17:26:07.621798 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.621689    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 17:26:07.622074 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.622057    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 17:26:07.631612 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.631239    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57db55b47c-9z44z"]
Sep 10 17:26:07.631726 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.631694    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 17:26:07.664297 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.664265    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-service-ca\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.664435 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.664325    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-trusted-ca-bundle\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.664435 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.664365    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-serving-cert\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.664435 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.664418    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-config\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.664605 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.664461    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-oauth-config\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.664605 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.664485    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-oauth-serving-cert\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.664605 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.664576    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms874\" (UniqueName: \"kubernetes.io/projected/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-kube-api-access-ms874\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.765196 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.765157    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-oauth-config\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.765355 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.765214    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-oauth-serving-cert\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.765355 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.765256    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms874\" (UniqueName: \"kubernetes.io/projected/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-kube-api-access-ms874\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.765355 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.765296    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-service-ca\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.765659 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.765627    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-trusted-ca-bundle\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.765765 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.765691    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-serving-cert\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.765765 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.765725    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-config\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.766106 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.766076    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-service-ca\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.766201 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.766076    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-oauth-serving-cert\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.766424 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.766399    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-config\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.766538 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.766486    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-trusted-ca-bundle\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.768177 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.768154    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-oauth-config\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.768969 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.768948    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-serving-cert\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.775450 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.775428    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms874\" (UniqueName: \"kubernetes.io/projected/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-kube-api-access-ms874\") pod \"console-57db55b47c-9z44z\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") " pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:07.928216 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:07.928108    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:08.206225 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:08.206133    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57db55b47c-9z44z"]
Sep 10 17:26:08.209511 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:26:08.209464    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b867cf2_06f8_4e7b_a903_b1c2f3ff7d52.slice/crio-8afc7b9353e7fc7b994abe609caf1a6f7adb5137b7ee1aa3b4b445ed8ddf320a WatchSource:0}: Error finding container 8afc7b9353e7fc7b994abe609caf1a6f7adb5137b7ee1aa3b4b445ed8ddf320a: Status 404 returned error can't find the container with id 8afc7b9353e7fc7b994abe609caf1a6f7adb5137b7ee1aa3b4b445ed8ddf320a
Sep 10 17:26:08.464239 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:08.464154    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57db55b47c-9z44z" event={"ID":"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52","Type":"ContainerStarted","Data":"8afc7b9353e7fc7b994abe609caf1a6f7adb5137b7ee1aa3b4b445ed8ddf320a"}
Sep 10 17:26:11.577436 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.577351    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mlz6w"]
Sep 10 17:26:11.580144 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.580088    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.583004 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.582898    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v58mw\""
Sep 10 17:26:11.583004 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.582911    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 17:26:11.584119 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.583872    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 17:26:11.584119 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.583954    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 17:26:11.584275 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.584196    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 17:26:11.584381 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.584347    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 17:26:11.584381 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.584374    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 17:26:11.597252 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.597223    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-textfile\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.597368 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.597265    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-tls\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.597368 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.597291    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-wtmp\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.597368 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.597338    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/85a41d92-3f00-4000-96a2-9d49c60196be-root\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.597605 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.597371    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.597605 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.597397    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85a41d92-3f00-4000-96a2-9d49c60196be-metrics-client-ca\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.597605 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.597427    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85a41d92-3f00-4000-96a2-9d49c60196be-sys\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.597605 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.597465    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-accelerators-collector-config\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.597605 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.597584    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9cln\" (UniqueName: \"kubernetes.io/projected/85a41d92-3f00-4000-96a2-9d49c60196be-kube-api-access-n9cln\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.698429 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.698386    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-wtmp\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.698611 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.698463    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/85a41d92-3f00-4000-96a2-9d49c60196be-root\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.698611 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.698514    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.698611 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.698542    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85a41d92-3f00-4000-96a2-9d49c60196be-metrics-client-ca\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.698611 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.698574    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85a41d92-3f00-4000-96a2-9d49c60196be-sys\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.698611 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.698584    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-wtmp\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.698870 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.698614    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-accelerators-collector-config\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.698870 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.698668    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9cln\" (UniqueName: \"kubernetes.io/projected/85a41d92-3f00-4000-96a2-9d49c60196be-kube-api-access-n9cln\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.698870 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.698703    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-textfile\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.698870 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.698727    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-tls\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.698870 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.698747    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/85a41d92-3f00-4000-96a2-9d49c60196be-root\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.698870 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:26:11.698834    2552 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 17:26:11.699145 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:26:11.698891    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-tls podName:85a41d92-3f00-4000-96a2-9d49c60196be nodeName:}" failed. No retries permitted until 2026-09-10 17:26:12.198872274 +0000 UTC m=+143.957399326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-tls") pod "node-exporter-mlz6w" (UID: "85a41d92-3f00-4000-96a2-9d49c60196be") : secret "node-exporter-tls" not found
Sep 10 17:26:11.699745 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.699230    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-textfile\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.699745 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.699291    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85a41d92-3f00-4000-96a2-9d49c60196be-sys\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.699745 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.699332    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-accelerators-collector-config\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.699745 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.699705    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85a41d92-3f00-4000-96a2-9d49c60196be-metrics-client-ca\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.708538 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.701597    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:11.719619 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:11.719592    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9cln\" (UniqueName: \"kubernetes.io/projected/85a41d92-3f00-4000-96a2-9d49c60196be-kube-api-access-n9cln\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:12.203611 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.203569    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-tls\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:12.206206 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.206175    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/85a41d92-3f00-4000-96a2-9d49c60196be-node-exporter-tls\") pod \"node-exporter-mlz6w\" (UID: \"85a41d92-3f00-4000-96a2-9d49c60196be\") " pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:12.493456 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.493383    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mlz6w"
Sep 10 17:26:12.586895 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.586858    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:26:12.590024 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.589999    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.592898 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.592873    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 17:26:12.593014 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.592904    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 17:26:12.593014 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.592904    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 17:26:12.593309 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.593228    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 17:26:12.593409 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.593230    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 17:26:12.593564 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.593523    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 17:26:12.593811 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.593790    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 17:26:12.593887 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.593791    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 17:26:12.593887 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.593795    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 17:26:12.593887 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.593795    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-d95xn\""
Sep 10 17:26:12.607174 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.607146    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:26:12.708360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.708321    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.708561 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.708385    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db3f88aa-1a56-4180-852f-0710838303d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.708561 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.708483    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-web-config\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.708693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.708589    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db3f88aa-1a56-4180-852f-0710838303d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.708693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.708626    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.708693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.708660    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.708839 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.708720    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db3f88aa-1a56-4180-852f-0710838303d4-config-out\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.708839 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.708756    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.708839 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.708790    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db3f88aa-1a56-4180-852f-0710838303d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.708976 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.708833    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.708976 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.708866    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db3f88aa-1a56-4180-852f-0710838303d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.708976 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.708893    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.708976 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.708919    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsd99\" (UniqueName: \"kubernetes.io/projected/db3f88aa-1a56-4180-852f-0710838303d4-kube-api-access-jsd99\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.810198 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.810109    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-web-config\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.810198 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.810175    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db3f88aa-1a56-4180-852f-0710838303d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.810400 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.810201    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.810400 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.810226    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.810400 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.810255    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db3f88aa-1a56-4180-852f-0710838303d4-config-out\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.810400 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.810283    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.810400 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.810316    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db3f88aa-1a56-4180-852f-0710838303d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.810705 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.810459    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.810705 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.810508    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db3f88aa-1a56-4180-852f-0710838303d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.810705 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.810557    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.810705 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.810583    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsd99\" (UniqueName: \"kubernetes.io/projected/db3f88aa-1a56-4180-852f-0710838303d4-kube-api-access-jsd99\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.810705 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.810617    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.810705 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.810668    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db3f88aa-1a56-4180-852f-0710838303d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.813930 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.812599    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db3f88aa-1a56-4180-852f-0710838303d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.813930 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.812896    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db3f88aa-1a56-4180-852f-0710838303d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.813930 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.813770    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db3f88aa-1a56-4180-852f-0710838303d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.813930 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.813877    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.814261 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.814236    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.814664 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.814641    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.814763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.814667    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.815007 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.814929    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db3f88aa-1a56-4180-852f-0710838303d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.815170 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.815143    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-web-config\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.815696 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.815650    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.816067 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.816045    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db3f88aa-1a56-4180-852f-0710838303d4-config-out\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.816596 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.816570    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.819527 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.819486    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsd99\" (UniqueName: \"kubernetes.io/projected/db3f88aa-1a56-4180-852f-0710838303d4-kube-api-access-jsd99\") pod \"alertmanager-main-0\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:12.901630 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:12.901591    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:26:14.504078 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.504042    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-84f9766557-xzzlt"]
Sep 10 17:26:14.507350 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.507319    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.510694 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.510399    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 10 17:26:14.510694 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.510414    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 10 17:26:14.510694 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.510464    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 10 17:26:14.510921 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.510727    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 10 17:26:14.510921 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.510842    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-r45qb\""
Sep 10 17:26:14.510921 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.510894    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 10 17:26:14.511172 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.511154    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6egkhqrvlcro0\""
Sep 10 17:26:14.519958 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.519924    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84f9766557-xzzlt"]
Sep 10 17:26:14.625385 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.625352    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxs8f\" (UniqueName: \"kubernetes.io/projected/fd3cb3c3-0799-45d5-8a45-776823e46cec-kube-api-access-kxs8f\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.625577 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.625409    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-thanos-querier-tls\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.625577 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.625454    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-grpc-tls\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.625577 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.625563    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.625743 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.625640    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.625743 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.625715    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.625812 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.625774    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.625812 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.625801    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cb3c3-0799-45d5-8a45-776823e46cec-metrics-client-ca\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.727306 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.727262    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxs8f\" (UniqueName: \"kubernetes.io/projected/fd3cb3c3-0799-45d5-8a45-776823e46cec-kube-api-access-kxs8f\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.727532 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.727332    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-thanos-querier-tls\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.727532 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.727384    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-grpc-tls\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.727532 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.727440    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.727532 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.727475    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.727762 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.727545    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.727762 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.727584    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.727762 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.727609    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cb3c3-0799-45d5-8a45-776823e46cec-metrics-client-ca\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.728430 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.728397    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cb3c3-0799-45d5-8a45-776823e46cec-metrics-client-ca\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.730832 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.730788    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-grpc-tls\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.730944 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.730929    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-thanos-querier-tls\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.731008 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.730926    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.731008 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.730995    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.731115 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.731082    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.731170 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.731108    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd3cb3c3-0799-45d5-8a45-776823e46cec-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.735980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.735949    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxs8f\" (UniqueName: \"kubernetes.io/projected/fd3cb3c3-0799-45d5-8a45-776823e46cec-kube-api-access-kxs8f\") pod \"thanos-querier-84f9766557-xzzlt\" (UID: \"fd3cb3c3-0799-45d5-8a45-776823e46cec\") " pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:14.818772 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:14.818678    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:15.833590 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.833557    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-869f7667f5-qbtf8"]
Sep 10 17:26:15.836031 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.836009    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:15.838793 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.838769    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 17:26:15.840391 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.840364    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nzhp4\""
Sep 10 17:26:15.840391 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.840378    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 10 17:26:15.840391 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.840385    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 10 17:26:15.840680 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.840407    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 10 17:26:15.840680 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.840634    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-40og6mbd3r8l8\""
Sep 10 17:26:15.847238 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.847217    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-869f7667f5-qbtf8"]
Sep 10 17:26:15.938549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.938508    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7312f60e-373d-49fa-8213-c609144cd5df-secret-metrics-server-client-certs\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:15.938745 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.938565    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7312f60e-373d-49fa-8213-c609144cd5df-audit-log\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:15.938745 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.938597    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7312f60e-373d-49fa-8213-c609144cd5df-client-ca-bundle\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:15.938745 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.938629    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7312f60e-373d-49fa-8213-c609144cd5df-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:15.938745 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.938718    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7312f60e-373d-49fa-8213-c609144cd5df-secret-metrics-server-tls\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:15.938937 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.938756    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwqkd\" (UniqueName: \"kubernetes.io/projected/7312f60e-373d-49fa-8213-c609144cd5df-kube-api-access-gwqkd\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:15.938937 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:15.938805    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7312f60e-373d-49fa-8213-c609144cd5df-metrics-server-audit-profiles\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:16.040272 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:16.040234    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7312f60e-373d-49fa-8213-c609144cd5df-secret-metrics-server-tls\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:16.040272 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:16.040287    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwqkd\" (UniqueName: \"kubernetes.io/projected/7312f60e-373d-49fa-8213-c609144cd5df-kube-api-access-gwqkd\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:16.040529 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:16.040319    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7312f60e-373d-49fa-8213-c609144cd5df-metrics-server-audit-profiles\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:16.040529 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:16.040433    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7312f60e-373d-49fa-8213-c609144cd5df-secret-metrics-server-client-certs\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:16.040529 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:16.040465    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7312f60e-373d-49fa-8213-c609144cd5df-audit-log\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:16.040529 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:16.040515    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7312f60e-373d-49fa-8213-c609144cd5df-client-ca-bundle\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:16.040762 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:16.040733    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7312f60e-373d-49fa-8213-c609144cd5df-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:16.040956 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:16.040905    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7312f60e-373d-49fa-8213-c609144cd5df-audit-log\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:16.041458 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:16.041402    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7312f60e-373d-49fa-8213-c609144cd5df-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:16.041458 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:16.041445    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7312f60e-373d-49fa-8213-c609144cd5df-metrics-server-audit-profiles\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:16.043378 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:16.043331    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7312f60e-373d-49fa-8213-c609144cd5df-secret-metrics-server-tls\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:16.043590 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:16.043573    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7312f60e-373d-49fa-8213-c609144cd5df-secret-metrics-server-client-certs\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:16.043665 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:16.043630    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7312f60e-373d-49fa-8213-c609144cd5df-client-ca-bundle\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:16.048633 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:16.048608    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwqkd\" (UniqueName: \"kubernetes.io/projected/7312f60e-373d-49fa-8213-c609144cd5df-kube-api-access-gwqkd\") pod \"metrics-server-869f7667f5-qbtf8\" (UID: \"7312f60e-373d-49fa-8213-c609144cd5df\") " pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:16.147143 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:16.147096    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:17.714508 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.714450    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:26:17.721147 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.720805    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.723910 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.723881    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 17:26:17.724059 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.723882    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 17:26:17.724059 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.723943    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5asi0i0iho0uo\""
Sep 10 17:26:17.724059 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.723952    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 17:26:17.724059 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.723892    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 17:26:17.724838 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.724802    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 17:26:17.724951 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.724841    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 17:26:17.724951 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.724848    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 17:26:17.724951 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.724802    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 17:26:17.725184 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.725165    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-srxdj\""
Sep 10 17:26:17.725250 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.725230    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 17:26:17.725310 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.725275    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 17:26:17.726171 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.726150    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 17:26:17.726418 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.726362    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 17:26:17.726418 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.726385    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 17:26:17.728194 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.728172    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 17:26:17.734922 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.734898    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:26:17.756421 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.756385    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.756580 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.756435    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d905844a-59bb-494a-bd3e-1f1eee385869-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.756580 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.756522    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.756580 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.756559    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.756753 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.756584    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.756753 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.756610    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.756753 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.756640    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.756753 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.756699    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.756753 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.756726    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.757027 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.756777    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.757027 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.756838    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.757027 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.756875    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.757027 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.756911    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.757027 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.756954    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.757027 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.756979    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-config\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.757027 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.757004    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.757375 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.757120    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.757375 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.757150    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d905844a-59bb-494a-bd3e-1f1eee385869-config-out\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.757375 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.757189    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbd5x\" (UniqueName: \"kubernetes.io/projected/d905844a-59bb-494a-bd3e-1f1eee385869-kube-api-access-pbd5x\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.757375 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.757223    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-web-config\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.857895 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.857856    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.857895 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.857900    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-config\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858135 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.857927    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858135 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.857962    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858135 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.857984    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d905844a-59bb-494a-bd3e-1f1eee385869-config-out\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858994 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.858463    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbd5x\" (UniqueName: \"kubernetes.io/projected/d905844a-59bb-494a-bd3e-1f1eee385869-kube-api-access-pbd5x\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858994 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.858542    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-web-config\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858994 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.858591    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858994 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.858619    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d905844a-59bb-494a-bd3e-1f1eee385869-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858994 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.858655    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858994 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.858678    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858994 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.858705    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858994 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.858726    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858994 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.858754    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858994 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.858812    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858994 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.858840    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858994 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.858888    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858994 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.858919    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.858994 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.858949    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.860296 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.859232    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.860296 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.859262    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.860296 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.859401    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.861611 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.860795    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d905844a-59bb-494a-bd3e-1f1eee385869-config-out\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.861611 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.861539    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.862362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.862034    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-config\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.862362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.862081    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.862362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.862092    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.862362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.862112    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.862959 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.862925    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.863061 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.862956    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.863061 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.862977    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.863061 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.862998    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.863243 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.863220    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.863804 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.863755    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.864394 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.864345    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.864926 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.864883    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.865379 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.865359    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.865547 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.865512    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-web-config\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.865912 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.865887    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d905844a-59bb-494a-bd3e-1f1eee385869-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:17.876269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:17.876251    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbd5x\" (UniqueName: \"kubernetes.io/projected/d905844a-59bb-494a-bd3e-1f1eee385869-kube-api-access-pbd5x\") pod \"prometheus-k8s-0\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:18.032745 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.032664    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:18.054700 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:26:18.054661    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a41d92_3f00_4000_96a2_9d49c60196be.slice/crio-71c9c1fc21c6d44466630195528c2f8e6955c958b922b25eeadfc511b06dd4a8 WatchSource:0}: Error finding container 71c9c1fc21c6d44466630195528c2f8e6955c958b922b25eeadfc511b06dd4a8: Status 404 returned error can't find the container with id 71c9c1fc21c6d44466630195528c2f8e6955c958b922b25eeadfc511b06dd4a8
Sep 10 17:26:18.334970 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.334933    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-869f7667f5-qbtf8"]
Sep 10 17:26:18.338270 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:26:18.338237    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7312f60e_373d_49fa_8213_c609144cd5df.slice/crio-cb384a973b1b104ff019b0650696f5b01de9105b5acbe78bb1022e81c7f1edd8 WatchSource:0}: Error finding container cb384a973b1b104ff019b0650696f5b01de9105b5acbe78bb1022e81c7f1edd8: Status 404 returned error can't find the container with id cb384a973b1b104ff019b0650696f5b01de9105b5acbe78bb1022e81c7f1edd8
Sep 10 17:26:18.339554 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.339532    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84f9766557-xzzlt"]
Sep 10 17:26:18.342484 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:26:18.342453    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3cb3c3_0799_45d5_8a45_776823e46cec.slice/crio-04fff510cc5d016b93a55a213854fab87ddcdf9a14c1a27827099bac7573fe48 WatchSource:0}: Error finding container 04fff510cc5d016b93a55a213854fab87ddcdf9a14c1a27827099bac7573fe48: Status 404 returned error can't find the container with id 04fff510cc5d016b93a55a213854fab87ddcdf9a14c1a27827099bac7573fe48
Sep 10 17:26:18.359910 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.359798    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:26:18.362689 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.362605    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:26:18.364273 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:26:18.364240    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3f88aa_1a56_4180_852f_0710838303d4.slice/crio-11e467eb9d56f89ca2cc88631bd3e0d15bee1c8080fff01e2c7ca3a567d7e791 WatchSource:0}: Error finding container 11e467eb9d56f89ca2cc88631bd3e0d15bee1c8080fff01e2c7ca3a567d7e791: Status 404 returned error can't find the container with id 11e467eb9d56f89ca2cc88631bd3e0d15bee1c8080fff01e2c7ca3a567d7e791
Sep 10 17:26:18.365226 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:26:18.365196    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd905844a_59bb_494a_bd3e_1f1eee385869.slice/crio-cd2d13131fea88a2341530f85c2aa555bb41e3f494cdd8c7ed3d8d76db03d98b WatchSource:0}: Error finding container cd2d13131fea88a2341530f85c2aa555bb41e3f494cdd8c7ed3d8d76db03d98b: Status 404 returned error can't find the container with id cd2d13131fea88a2341530f85c2aa555bb41e3f494cdd8c7ed3d8d76db03d98b
Sep 10 17:26:18.511791 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.511749    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-ccl54" event={"ID":"59a60e4b-21b8-4ede-b64c-8164dc12742c","Type":"ContainerStarted","Data":"fbbe8bc94d35e82120d019587ee3a0c7843f4763bcf3ea68dbdfd085787816b9"}
Sep 10 17:26:18.512475 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.512429    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-ccl54"
Sep 10 17:26:18.516517 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.515878    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt" event={"ID":"fd3cb3c3-0799-45d5-8a45-776823e46cec","Type":"ContainerStarted","Data":"04fff510cc5d016b93a55a213854fab87ddcdf9a14c1a27827099bac7573fe48"}
Sep 10 17:26:18.519088 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.519060    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mlz6w" event={"ID":"85a41d92-3f00-4000-96a2-9d49c60196be","Type":"ContainerStarted","Data":"71c9c1fc21c6d44466630195528c2f8e6955c958b922b25eeadfc511b06dd4a8"}
Sep 10 17:26:18.521136 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.521093    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3f88aa-1a56-4180-852f-0710838303d4","Type":"ContainerStarted","Data":"11e467eb9d56f89ca2cc88631bd3e0d15bee1c8080fff01e2c7ca3a567d7e791"}
Sep 10 17:26:18.524891 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.524861    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57db55b47c-9z44z" event={"ID":"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52","Type":"ContainerStarted","Data":"3d72e4f6748bdd26d137e1011e4603c6da7e3260a5405e7bcf55c6a47af10bda"}
Sep 10 17:26:18.525087 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.525067    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-ccl54"
Sep 10 17:26:18.526528 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.526470    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8" event={"ID":"7312f60e-373d-49fa-8213-c609144cd5df","Type":"ContainerStarted","Data":"cb384a973b1b104ff019b0650696f5b01de9105b5acbe78bb1022e81c7f1edd8"}
Sep 10 17:26:18.528559 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.528464    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerStarted","Data":"cd2d13131fea88a2341530f85c2aa555bb41e3f494cdd8c7ed3d8d76db03d98b"}
Sep 10 17:26:18.532433 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.532382    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-ccl54" podStartSLOduration=1.350827918 podStartE2EDuration="21.532367715s" totalImagesPullingTime="20.181539797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:18.530850779 +0000 UTC m=+150.289377850" watchObservedRunningTime="2026-09-10 17:26:18.532367715 +0000 UTC m=+150.290894785"
Sep 10 17:26:18.570227 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:18.570082    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57db55b47c-9z44z" podStartSLOduration=1.726077699 podStartE2EDuration="11.570060969s" totalImagesPullingTime="9.84398327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:18.567917296 +0000 UTC m=+150.326444367" watchObservedRunningTime="2026-09-10 17:26:18.570060969 +0000 UTC m=+150.328588040"
Sep 10 17:26:19.540045 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:19.540008    2552 generic.go:356] "Generic (PLEG): container finished" podID="85a41d92-3f00-4000-96a2-9d49c60196be" containerID="83e4757b542762c9291b27bee20181362f446e7658a11349aaa061abaaf85cdc" exitCode=0
Sep 10 17:26:19.542205 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:19.541239    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mlz6w" event={"ID":"85a41d92-3f00-4000-96a2-9d49c60196be","Type":"ContainerDied","Data":"83e4757b542762c9291b27bee20181362f446e7658a11349aaa061abaaf85cdc"}
Sep 10 17:26:20.551087 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:20.550324    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mlz6w" event={"ID":"85a41d92-3f00-4000-96a2-9d49c60196be","Type":"ContainerStarted","Data":"8b707e37e55c802282b453793db2d3936618c9afab019e0e3fa07abb9b196b53"}
Sep 10 17:26:20.551087 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:20.550465    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mlz6w" event={"ID":"85a41d92-3f00-4000-96a2-9d49c60196be","Type":"ContainerStarted","Data":"ab6828657eff7b3142a6c798fe6af5bc0cd6e3284d7f3958e90f00860a723ded"}
Sep 10 17:26:20.577516 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:20.574351    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mlz6w" podStartSLOduration=8.542070457 podStartE2EDuration="9.574330669s" totalImagesPullingTime="930.029597ms" totalInitContainerRuntime="102.230615ms" isStatefulPod=true podCreationTimestamp="2026-09-10 17:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:20.572515609 +0000 UTC m=+152.331042680" watchObservedRunningTime="2026-09-10 17:26:20.574330669 +0000 UTC m=+152.332857742"
Sep 10 17:26:22.146208 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.146168    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57db55b47c-9z44z"]
Sep 10 17:26:22.187393 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.187357    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8c8ffbcdb-mmqhc"]
Sep 10 17:26:22.209742 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.209703    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8c8ffbcdb-mmqhc"]
Sep 10 17:26:22.209915 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.209829    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.313034 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.312981    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-trusted-ca-bundle\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.313183 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.313142    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a38850ba-1f03-4937-8948-07de844bc2eb-console-serving-cert\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.313183 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.313170    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjlk9\" (UniqueName: \"kubernetes.io/projected/a38850ba-1f03-4937-8948-07de844bc2eb-kube-api-access-qjlk9\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.313304 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.313222    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a38850ba-1f03-4937-8948-07de844bc2eb-console-oauth-config\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.313304 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.313245    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-console-config\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.313304 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.313281    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-service-ca\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.313433 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.313362    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-oauth-serving-cert\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.414592 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.414559    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a38850ba-1f03-4937-8948-07de844bc2eb-console-serving-cert\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.414722 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.414602    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjlk9\" (UniqueName: \"kubernetes.io/projected/a38850ba-1f03-4937-8948-07de844bc2eb-kube-api-access-qjlk9\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.414722 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.414647    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a38850ba-1f03-4937-8948-07de844bc2eb-console-oauth-config\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.414722 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.414685    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-console-config\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.414885 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.414809    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-service-ca\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.414885 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.414869    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-oauth-serving-cert\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.414987 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.414905    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-trusted-ca-bundle\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.415758 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.415677    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-console-config\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.415928 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.415883    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-service-ca\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.416143 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.416112    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-trusted-ca-bundle\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.416591 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.416570    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-oauth-serving-cert\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.417698 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.417675    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a38850ba-1f03-4937-8948-07de844bc2eb-console-oauth-config\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.417953 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.417933    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a38850ba-1f03-4937-8948-07de844bc2eb-console-serving-cert\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.423999 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.423953    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjlk9\" (UniqueName: \"kubernetes.io/projected/a38850ba-1f03-4937-8948-07de844bc2eb-kube-api-access-qjlk9\") pod \"console-8c8ffbcdb-mmqhc\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") " pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.520853 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.520688    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:22.569308 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.569174    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8" event={"ID":"7312f60e-373d-49fa-8213-c609144cd5df","Type":"ContainerStarted","Data":"fc035513129c05d38f6629a3dd69717c019b3274e90c8d4e766debc44df1f7f2"}
Sep 10 17:26:22.574970 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.573256    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerStarted","Data":"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"}
Sep 10 17:26:22.592596 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.592538    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8" podStartSLOduration=3.534701933 podStartE2EDuration="7.592522379s" totalImagesPullingTime="4.057820446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:22.590227516 +0000 UTC m=+154.348754611" watchObservedRunningTime="2026-09-10 17:26:22.592522379 +0000 UTC m=+154.351049463"
Sep 10 17:26:22.842688 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:22.842664    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8c8ffbcdb-mmqhc"]
Sep 10 17:26:22.846703 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:26:22.846675    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38850ba_1f03_4937_8948_07de844bc2eb.slice/crio-cebaae6eab2e2c41e0b8d9d77bc9a287f3746809cbf54d7f5c4ae3f034629615 WatchSource:0}: Error finding container cebaae6eab2e2c41e0b8d9d77bc9a287f3746809cbf54d7f5c4ae3f034629615: Status 404 returned error can't find the container with id cebaae6eab2e2c41e0b8d9d77bc9a287f3746809cbf54d7f5c4ae3f034629615
Sep 10 17:26:23.579710 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:23.579674    2552 generic.go:356] "Generic (PLEG): container finished" podID="d905844a-59bb-494a-bd3e-1f1eee385869" containerID="97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0" exitCode=0
Sep 10 17:26:23.580171 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:23.579754    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerDied","Data":"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"}
Sep 10 17:26:23.583433 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:23.582781    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt" event={"ID":"fd3cb3c3-0799-45d5-8a45-776823e46cec","Type":"ContainerStarted","Data":"8caa7c8d3297c4a18e654daa1151a1c48202ed0c66f2a4ba339b3ae282d3e71a"}
Sep 10 17:26:23.583433 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:23.582858    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt" event={"ID":"fd3cb3c3-0799-45d5-8a45-776823e46cec","Type":"ContainerStarted","Data":"a90781286e95adcb5523230982f91b45e581fc69a2c18671c7f5df3a07ca30b1"}
Sep 10 17:26:23.583433 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:23.582898    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt" event={"ID":"fd3cb3c3-0799-45d5-8a45-776823e46cec","Type":"ContainerStarted","Data":"0cf3f91bace119ab10a5c5f236af5861215b1161b0e70b14086494a373a509f7"}
Sep 10 17:26:23.585436 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:23.585405    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c8ffbcdb-mmqhc" event={"ID":"a38850ba-1f03-4937-8948-07de844bc2eb","Type":"ContainerStarted","Data":"1b0837907c008c4627848d0bf4d6a358ab648cfafde00eced96af0512b8d7a6f"}
Sep 10 17:26:23.585557 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:23.585464    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c8ffbcdb-mmqhc" event={"ID":"a38850ba-1f03-4937-8948-07de844bc2eb","Type":"ContainerStarted","Data":"cebaae6eab2e2c41e0b8d9d77bc9a287f3746809cbf54d7f5c4ae3f034629615"}
Sep 10 17:26:23.587417 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:23.587393    2552 generic.go:356] "Generic (PLEG): container finished" podID="db3f88aa-1a56-4180-852f-0710838303d4" containerID="c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690" exitCode=0
Sep 10 17:26:23.587601 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:23.587573    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3f88aa-1a56-4180-852f-0710838303d4","Type":"ContainerDied","Data":"c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690"}
Sep 10 17:26:23.632804 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:23.631799    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8c8ffbcdb-mmqhc" podStartSLOduration=1.6317747580000002 podStartE2EDuration="1.631774758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:23.630983191 +0000 UTC m=+155.389510262" watchObservedRunningTime="2026-09-10 17:26:23.631774758 +0000 UTC m=+155.390301829"
Sep 10 17:26:24.601339 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:24.601302    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt" event={"ID":"fd3cb3c3-0799-45d5-8a45-776823e46cec","Type":"ContainerStarted","Data":"e25e6975b5b8972062bae550e8a08551150e73587dbe4e0287cd5fe03566a6cc"}
Sep 10 17:26:25.614487 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:25.614412    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt" event={"ID":"fd3cb3c3-0799-45d5-8a45-776823e46cec","Type":"ContainerStarted","Data":"02f17c8e59f8a4f6af16dcb2720c97cdcbeaaa6ffeb554f021ab46d9e45f1bab"}
Sep 10 17:26:25.615280 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:25.615250    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt" event={"ID":"fd3cb3c3-0799-45d5-8a45-776823e46cec","Type":"ContainerStarted","Data":"e4f344b5974a126caed91d22e47a4efaff4fbccf064f3355c9ebf04da52a5edf"}
Sep 10 17:26:25.615406 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:25.615316    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:25.639241 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:25.639138    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt" podStartSLOduration=6.047614608 podStartE2EDuration="11.639115351s" totalImagesPullingTime="5.591500743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:25.636447867 +0000 UTC m=+157.394974941" watchObservedRunningTime="2026-09-10 17:26:25.639115351 +0000 UTC m=+157.397642422"
Sep 10 17:26:25.666592 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:26:25.666537    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl" podUID="47b034f8-d52d-40ca-9c28-4e2931eb61b3"
Sep 10 17:26:25.677858 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:26:25.677816    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-77x6z" podUID="a3b1670d-12c0-4614-a86e-22c5498e1274"
Sep 10 17:26:26.621806 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:26.621770    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-77x6z"
Sep 10 17:26:26.621806 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:26.621777    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3f88aa-1a56-4180-852f-0710838303d4","Type":"ContainerStarted","Data":"1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74"}
Sep 10 17:26:27.629090 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:27.629049    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3f88aa-1a56-4180-852f-0710838303d4","Type":"ContainerStarted","Data":"84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1"}
Sep 10 17:26:27.928946 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:27.928919    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:28.638364 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:28.638330    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3f88aa-1a56-4180-852f-0710838303d4","Type":"ContainerStarted","Data":"29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0"}
Sep 10 17:26:28.638678 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:28.638453    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3f88aa-1a56-4180-852f-0710838303d4","Type":"ContainerStarted","Data":"4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a"}
Sep 10 17:26:28.638678 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:28.638514    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3f88aa-1a56-4180-852f-0710838303d4","Type":"ContainerStarted","Data":"1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522"}
Sep 10 17:26:28.638678 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:28.638556    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3f88aa-1a56-4180-852f-0710838303d4","Type":"ContainerStarted","Data":"3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3"}
Sep 10 17:26:28.641749 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:28.641727    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerStarted","Data":"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"}
Sep 10 17:26:28.641841 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:28.641783    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerStarted","Data":"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"}
Sep 10 17:26:28.667075 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:28.667017    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=9.762524672 podStartE2EDuration="16.666995499s" totalImagesPullingTime="6.895075575s" totalInitContainerRuntime="9.395252ms" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:28.66602676 +0000 UTC m=+160.424553834" watchObservedRunningTime="2026-09-10 17:26:28.666995499 +0000 UTC m=+160.425522571"
Sep 10 17:26:29.649820 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:29.649780    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerStarted","Data":"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"}
Sep 10 17:26:29.650301 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:29.649868    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerStarted","Data":"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"}
Sep 10 17:26:29.650301 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:29.649912    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerStarted","Data":"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"}
Sep 10 17:26:29.650301 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:29.649955    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerStarted","Data":"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"}
Sep 10 17:26:29.680744 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:29.680694    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.876279215 podStartE2EDuration="12.680675452s" totalImagesPullingTime="8.761379356s" totalInitContainerRuntime="43.016881ms" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:29.678108121 +0000 UTC m=+161.436635212" watchObservedRunningTime="2026-09-10 17:26:29.680675452 +0000 UTC m=+161.439202522"
Sep 10 17:26:30.627791 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:30.627749    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:26:30.628034 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:30.627804    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:26:30.630178 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:30.630148    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3b1670d-12c0-4614-a86e-22c5498e1274-metrics-tls\") pod \"dns-default-77x6z\" (UID: \"a3b1670d-12c0-4614-a86e-22c5498e1274\") " pod="openshift-dns/dns-default-77x6z"
Sep 10 17:26:30.630321 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:30.630298    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls\") pod \"image-registry-64f498f4fc-7k7bl\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") " pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:26:30.826121 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:30.826091    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vn8bp\""
Sep 10 17:26:30.833794 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:30.833753    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-77x6z"
Sep 10 17:26:31.123545 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:31.123518    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-77x6z"]
Sep 10 17:26:31.125524 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:26:31.125480    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b1670d_12c0_4614_a86e_22c5498e1274.slice/crio-003cf4caf4e9daa138c73a2ec4455abef7e9933523e637c599f7e580b6bbbda8 WatchSource:0}: Error finding container 003cf4caf4e9daa138c73a2ec4455abef7e9933523e637c599f7e580b6bbbda8: Status 404 returned error can't find the container with id 003cf4caf4e9daa138c73a2ec4455abef7e9933523e637c599f7e580b6bbbda8
Sep 10 17:26:31.628309 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:31.628283    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84f9766557-xzzlt"
Sep 10 17:26:31.660232 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:31.660187    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-77x6z" event={"ID":"a3b1670d-12c0-4614-a86e-22c5498e1274","Type":"ContainerStarted","Data":"003cf4caf4e9daa138c73a2ec4455abef7e9933523e637c599f7e580b6bbbda8"}
Sep 10 17:26:32.521288 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:32.521254    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:32.521765 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:32.521326    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:32.522835 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:32.522516    2552 patch_prober.go:28] interesting pod/console-8c8ffbcdb-mmqhc container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused" start-of-body=
Sep 10 17:26:32.522835 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:32.522583    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-8c8ffbcdb-mmqhc" podUID="a38850ba-1f03-4937-8948-07de844bc2eb" containerName="console" probeResult="failure" output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused"
Sep 10 17:26:33.033617 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:33.033576    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:26:34.681971 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:34.681933    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-77x6z" event={"ID":"a3b1670d-12c0-4614-a86e-22c5498e1274","Type":"ContainerStarted","Data":"2d6821cf17a10442612ba0fb535c79acdaa7aa51f4c532d5acc8e22f9da5b8fe"}
Sep 10 17:26:34.682362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:34.682025    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-77x6z" event={"ID":"a3b1670d-12c0-4614-a86e-22c5498e1274","Type":"ContainerStarted","Data":"cc61347b5f0e0099cbb0ca5dcacf65ce953f999faff69752367163c45c6e5933"}
Sep 10 17:26:34.682362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:34.682072    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-77x6z"
Sep 10 17:26:34.703510 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:34.703439    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-77x6z" podStartSLOduration=130.225411167 podStartE2EDuration="2m12.703424298s" totalImagesPullingTime="2.478013131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:34.702076742 +0000 UTC m=+166.460603846" watchObservedRunningTime="2026-09-10 17:26:34.703424298 +0000 UTC m=+166.461951368"
Sep 10 17:26:36.147713 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:36.147672    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:36.148096 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:36.147743    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:37.699879 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:37.699839    2552 generic.go:356] "Generic (PLEG): container finished" podID="846b39a3-9507-423f-ba80-1bbff55a4e44" containerID="2c4f2c19ecda3cc86590d95b44a8b2f766c37788c36a78d30761c28d1c0bac51" exitCode=0
Sep 10 17:26:37.700363 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:37.699907    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn" event={"ID":"846b39a3-9507-423f-ba80-1bbff55a4e44","Type":"ContainerDied","Data":"2c4f2c19ecda3cc86590d95b44a8b2f766c37788c36a78d30761c28d1c0bac51"}
Sep 10 17:26:37.700363 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:37.700322    2552 scope.go:117] "RemoveContainer" containerID="2c4f2c19ecda3cc86590d95b44a8b2f766c37788c36a78d30761c28d1c0bac51"
Sep 10 17:26:37.797448 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:37.797424    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:26:37.803389 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:37.803366    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kwhpq\""
Sep 10 17:26:37.807978 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:37.807957    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:26:38.103306 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:26:38.103267    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b034f8_d52d_40ca_9c28_4e2931eb61b3.slice/crio-45f454532e77a07a69238b97f668d7a3854d328c94f8ea5be47c1c52eda47afc WatchSource:0}: Error finding container 45f454532e77a07a69238b97f668d7a3854d328c94f8ea5be47c1c52eda47afc: Status 404 returned error can't find the container with id 45f454532e77a07a69238b97f668d7a3854d328c94f8ea5be47c1c52eda47afc
Sep 10 17:26:38.105113 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:38.105088    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64f498f4fc-7k7bl"]
Sep 10 17:26:38.705587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:38.705551    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jh5dn" event={"ID":"846b39a3-9507-423f-ba80-1bbff55a4e44","Type":"ContainerStarted","Data":"2ca815505924b44e393bd4a65ab643da17a17f620c1d238c13b51422a00c60dd"}
Sep 10 17:26:38.707108 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:38.707079    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl" event={"ID":"47b034f8-d52d-40ca-9c28-4e2931eb61b3","Type":"ContainerStarted","Data":"662201824d9cc5fb7fb86ccc7a582f36d27acce201071b95e9a38d15dd8542f8"}
Sep 10 17:26:38.707236 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:38.707124    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl" event={"ID":"47b034f8-d52d-40ca-9c28-4e2931eb61b3","Type":"ContainerStarted","Data":"45f454532e77a07a69238b97f668d7a3854d328c94f8ea5be47c1c52eda47afc"}
Sep 10 17:26:38.707236 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:38.707179    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:26:38.750583 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:38.750525    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl" podStartSLOduration=155.750489223 podStartE2EDuration="2m35.750489223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:38.750487109 +0000 UTC m=+170.509014182" watchObservedRunningTime="2026-09-10 17:26:38.750489223 +0000 UTC m=+170.509016294"
Sep 10 17:26:42.521325 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:42.521248    2552 patch_prober.go:28] interesting pod/console-8c8ffbcdb-mmqhc container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused" start-of-body=
Sep 10 17:26:42.521325 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:42.521306    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-8c8ffbcdb-mmqhc" podUID="a38850ba-1f03-4937-8948-07de844bc2eb" containerName="console" probeResult="failure" output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused"
Sep 10 17:26:44.688324 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:44.688289    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-77x6z"
Sep 10 17:26:47.168411 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:47.168343    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57db55b47c-9z44z" podUID="2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52" containerName="console" containerID="cri-o://3d72e4f6748bdd26d137e1011e4603c6da7e3260a5405e7bcf55c6a47af10bda" gracePeriod=15
Sep 10 17:26:48.520859 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.520834    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:48.597744 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.597717    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms874\" (UniqueName: \"kubernetes.io/projected/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-kube-api-access-ms874\") pod \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") "
Sep 10 17:26:48.597892 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.597789    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-service-ca\") pod \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") "
Sep 10 17:26:48.597892 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.597807    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-serving-cert\") pod \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") "
Sep 10 17:26:48.597989 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.597905    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-config\") pod \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") "
Sep 10 17:26:48.597989 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.597960    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-oauth-config\") pod \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") "
Sep 10 17:26:48.598153 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.598019    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-trusted-ca-bundle\") pod \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") "
Sep 10 17:26:48.598153 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.598062    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-oauth-serving-cert\") pod \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\" (UID: \"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52\") "
Sep 10 17:26:48.598308 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.598220    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-service-ca" (OuterVolumeSpecName: "service-ca") pod "2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52" (UID: "2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:26:48.598384 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.598373    2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:26:48.598441 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.598277    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-config" (OuterVolumeSpecName: "console-config") pod "2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52" (UID: "2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:26:48.598903 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.598481    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52" (UID: "2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:26:48.600558 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.600526    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52" (UID: "2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:26:48.600695 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.600672    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52" (UID: "2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:26:48.600695 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.600696    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-kube-api-access-ms874" (OuterVolumeSpecName: "kube-api-access-ms874") pod "2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52" (UID: "2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52"). InnerVolumeSpecName "kube-api-access-ms874". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:26:48.601059 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.601031    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52" (UID: "2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:26:48.698832 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.698799    2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-serving-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:26:48.698832 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.698828    2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-config\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:26:48.698832 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.698839    2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-console-oauth-config\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:26:48.699074 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.698848    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-trusted-ca-bundle\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:26:48.699074 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.698856    2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-oauth-serving-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:26:48.699074 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.698864    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms874\" (UniqueName: \"kubernetes.io/projected/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52-kube-api-access-ms874\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:26:48.749250 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.749211    2552 generic.go:356] "Generic (PLEG): container finished" podID="2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52" containerID="3d72e4f6748bdd26d137e1011e4603c6da7e3260a5405e7bcf55c6a47af10bda" exitCode=0
Sep 10 17:26:48.749438 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.749277    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57db55b47c-9z44z"
Sep 10 17:26:48.749438 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.749298    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57db55b47c-9z44z" event={"ID":"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52","Type":"ContainerDied","Data":"3d72e4f6748bdd26d137e1011e4603c6da7e3260a5405e7bcf55c6a47af10bda"}
Sep 10 17:26:48.749438 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.749389    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57db55b47c-9z44z" event={"ID":"2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52","Type":"ContainerDied","Data":"8afc7b9353e7fc7b994abe609caf1a6f7adb5137b7ee1aa3b4b445ed8ddf320a"}
Sep 10 17:26:48.749438 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.749418    2552 scope.go:117] "RemoveContainer" containerID="3d72e4f6748bdd26d137e1011e4603c6da7e3260a5405e7bcf55c6a47af10bda"
Sep 10 17:26:48.757788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.757770    2552 scope.go:117] "RemoveContainer" containerID="3d72e4f6748bdd26d137e1011e4603c6da7e3260a5405e7bcf55c6a47af10bda"
Sep 10 17:26:48.758049 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:26:48.758030    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d72e4f6748bdd26d137e1011e4603c6da7e3260a5405e7bcf55c6a47af10bda\": container with ID starting with 3d72e4f6748bdd26d137e1011e4603c6da7e3260a5405e7bcf55c6a47af10bda not found: ID does not exist" containerID="3d72e4f6748bdd26d137e1011e4603c6da7e3260a5405e7bcf55c6a47af10bda"
Sep 10 17:26:48.758120 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.758056    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d72e4f6748bdd26d137e1011e4603c6da7e3260a5405e7bcf55c6a47af10bda"} err="failed to get container status \"3d72e4f6748bdd26d137e1011e4603c6da7e3260a5405e7bcf55c6a47af10bda\": rpc error: code = NotFound desc = could not find container \"3d72e4f6748bdd26d137e1011e4603c6da7e3260a5405e7bcf55c6a47af10bda\": container with ID starting with 3d72e4f6748bdd26d137e1011e4603c6da7e3260a5405e7bcf55c6a47af10bda not found: ID does not exist"
Sep 10 17:26:48.771436 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.771399    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57db55b47c-9z44z"]
Sep 10 17:26:48.775678 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.775647    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57db55b47c-9z44z"]
Sep 10 17:26:48.801992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:48.801964    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52" path="/var/lib/kubelet/pods/2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52/volumes"
Sep 10 17:26:52.525302 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:52.525271    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:52.529100 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:52.529073    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:26:54.392651 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:54.392618    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db3f88aa-1a56-4180-852f-0710838303d4/init-config-reloader/0.log"
Sep 10 17:26:54.596701 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:54.596663    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db3f88aa-1a56-4180-852f-0710838303d4/alertmanager/0.log"
Sep 10 17:26:54.792354 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:54.792277    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db3f88aa-1a56-4180-852f-0710838303d4/config-reloader/0.log"
Sep 10 17:26:54.993954 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:54.993914    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db3f88aa-1a56-4180-852f-0710838303d4/kube-rbac-proxy-web/0.log"
Sep 10 17:26:55.193204 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:55.193167    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db3f88aa-1a56-4180-852f-0710838303d4/kube-rbac-proxy/0.log"
Sep 10 17:26:55.392847 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:55.392820    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db3f88aa-1a56-4180-852f-0710838303d4/kube-rbac-proxy-metric/0.log"
Sep 10 17:26:55.593089 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:55.593060    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db3f88aa-1a56-4180-852f-0710838303d4/prom-label-proxy/0.log"
Sep 10 17:26:56.153272 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:56.153238    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:56.157100 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:56.157077    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-869f7667f5-qbtf8"
Sep 10 17:26:56.591743 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:56.591713    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-869f7667f5-qbtf8_7312f60e-373d-49fa-8213-c609144cd5df/metrics-server/0.log"
Sep 10 17:26:57.592063 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:57.592038    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mlz6w_85a41d92-3f00-4000-96a2-9d49c60196be/init-textfile/0.log"
Sep 10 17:26:57.640920 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:57.640888    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64f498f4fc-7k7bl"]
Sep 10 17:26:57.645011 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:57.644986    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:26:57.793286 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:57.793244    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mlz6w_85a41d92-3f00-4000-96a2-9d49c60196be/node-exporter/0.log"
Sep 10 17:26:57.992163 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:57.992087    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mlz6w_85a41d92-3f00-4000-96a2-9d49c60196be/kube-rbac-proxy/0.log"
Sep 10 17:26:59.391934 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:59.391907    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d905844a-59bb-494a-bd3e-1f1eee385869/init-config-reloader/0.log"
Sep 10 17:26:59.592964 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:59.592928    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d905844a-59bb-494a-bd3e-1f1eee385869/prometheus/0.log"
Sep 10 17:26:59.798046 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:59.797969    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d905844a-59bb-494a-bd3e-1f1eee385869/config-reloader/0.log"
Sep 10 17:26:59.992517 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:26:59.992474    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d905844a-59bb-494a-bd3e-1f1eee385869/thanos-sidecar/0.log"
Sep 10 17:27:00.194663 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:00.194631    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d905844a-59bb-494a-bd3e-1f1eee385869/kube-rbac-proxy-web/0.log"
Sep 10 17:27:00.393139 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:00.393113    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d905844a-59bb-494a-bd3e-1f1eee385869/kube-rbac-proxy/0.log"
Sep 10 17:27:00.592860 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:00.592831    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d905844a-59bb-494a-bd3e-1f1eee385869/kube-rbac-proxy-thanos/0.log"
Sep 10 17:27:01.191335 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:01.191304    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-n9bcl_fb049c3e-b1ef-43b0-a466-18cd3d49575c/prometheus-operator-admission-webhook/0.log"
Sep 10 17:27:01.994104 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:01.994072    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9766557-xzzlt_fd3cb3c3-0799-45d5-8a45-776823e46cec/thanos-query/0.log"
Sep 10 17:27:02.192028 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:02.191996    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9766557-xzzlt_fd3cb3c3-0799-45d5-8a45-776823e46cec/kube-rbac-proxy-web/0.log"
Sep 10 17:27:02.392644 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:02.392613    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9766557-xzzlt_fd3cb3c3-0799-45d5-8a45-776823e46cec/kube-rbac-proxy/0.log"
Sep 10 17:27:02.593461 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:02.593423    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9766557-xzzlt_fd3cb3c3-0799-45d5-8a45-776823e46cec/prom-label-proxy/0.log"
Sep 10 17:27:02.793010 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:02.792923    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9766557-xzzlt_fd3cb3c3-0799-45d5-8a45-776823e46cec/kube-rbac-proxy-rules/0.log"
Sep 10 17:27:02.992604 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:02.992576    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9766557-xzzlt_fd3cb3c3-0799-45d5-8a45-776823e46cec/kube-rbac-proxy-metrics/0.log"
Sep 10 17:27:03.192279 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:03.192249    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-clnst_29057183-7c39-4679-80cd-aa4d4c2deebe/networking-console-plugin/0.log"
Sep 10 17:27:03.393993 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:03.393966    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:27:03.594427 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:03.594395    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/2.log"
Sep 10 17:27:03.993335 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:03.993249    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8c8ffbcdb-mmqhc_a38850ba-1f03-4937-8948-07de844bc2eb/console/0.log"
Sep 10 17:27:04.194160 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:04.194130    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-ccl54_59a60e4b-21b8-4ede-b64c-8164dc12742c/download-server/0.log"
Sep 10 17:27:04.392393 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:04.392341    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-77x6z_a3b1670d-12c0-4614-a86e-22c5498e1274/dns/0.log"
Sep 10 17:27:04.592225 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:04.592196    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-77x6z_a3b1670d-12c0-4614-a86e-22c5498e1274/kube-rbac-proxy/0.log"
Sep 10 17:27:05.991595 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:05.991570    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f9tg8_81246f30-ad68-472c-aa8b-72d14634f501/dns-node-resolver/0.log"
Sep 10 17:27:06.192906 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:06.192857    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-64f498f4fc-7k7bl_47b034f8-d52d-40ca-9c28-4e2931eb61b3/registry/0.log"
Sep 10 17:27:06.991877 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:06.991841    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wwbzr_7067d92a-5363-42fe-a8f0-7c765931c532/node-ca/0.log"
Sep 10 17:27:18.033447 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:18.033410    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:18.052951 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:18.052919    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:18.900756 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:18.900726    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:22.660932 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:22.660871    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl" podUID="47b034f8-d52d-40ca-9c28-4e2931eb61b3" containerName="registry" containerID="cri-o://662201824d9cc5fb7fb86ccc7a582f36d27acce201071b95e9a38d15dd8542f8" gracePeriod=30
Sep 10 17:27:23.904227 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:23.904198    2552 generic.go:356] "Generic (PLEG): container finished" podID="47b034f8-d52d-40ca-9c28-4e2931eb61b3" containerID="662201824d9cc5fb7fb86ccc7a582f36d27acce201071b95e9a38d15dd8542f8" exitCode=0
Sep 10 17:27:23.904565 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:23.904273    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl" event={"ID":"47b034f8-d52d-40ca-9c28-4e2931eb61b3","Type":"ContainerDied","Data":"662201824d9cc5fb7fb86ccc7a582f36d27acce201071b95e9a38d15dd8542f8"}
Sep 10 17:27:23.904565 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:23.904334    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl" event={"ID":"47b034f8-d52d-40ca-9c28-4e2931eb61b3","Type":"ContainerDied","Data":"45f454532e77a07a69238b97f668d7a3854d328c94f8ea5be47c1c52eda47afc"}
Sep 10 17:27:23.904565 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:23.904357    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45f454532e77a07a69238b97f668d7a3854d328c94f8ea5be47c1c52eda47afc"
Sep 10 17:27:23.912878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:23.912856    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:27:24.023644 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.023551    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-certificates\") pod \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") "
Sep 10 17:27:24.023644 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.023591    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47b034f8-d52d-40ca-9c28-4e2931eb61b3-trusted-ca\") pod \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") "
Sep 10 17:27:24.023644 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.023618    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/47b034f8-d52d-40ca-9c28-4e2931eb61b3-installation-pull-secrets\") pod \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") "
Sep 10 17:27:24.023908 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.023663    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-bound-sa-token\") pod \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") "
Sep 10 17:27:24.023908 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.023687    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knkm8\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-kube-api-access-knkm8\") pod \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") "
Sep 10 17:27:24.023908 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.023705    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/47b034f8-d52d-40ca-9c28-4e2931eb61b3-ca-trust-extracted\") pod \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") "
Sep 10 17:27:24.023908 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.023744    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls\") pod \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") "
Sep 10 17:27:24.023908 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.023772    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/47b034f8-d52d-40ca-9c28-4e2931eb61b3-image-registry-private-configuration\") pod \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\" (UID: \"47b034f8-d52d-40ca-9c28-4e2931eb61b3\") "
Sep 10 17:27:24.024152 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.023956    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "47b034f8-d52d-40ca-9c28-4e2931eb61b3" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:27:24.024152 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.024020    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b034f8-d52d-40ca-9c28-4e2931eb61b3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "47b034f8-d52d-40ca-9c28-4e2931eb61b3" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:27:24.026634 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.026605    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "47b034f8-d52d-40ca-9c28-4e2931eb61b3" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:27:24.026823 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.026797    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "47b034f8-d52d-40ca-9c28-4e2931eb61b3" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:27:24.026823 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.026808    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b034f8-d52d-40ca-9c28-4e2931eb61b3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "47b034f8-d52d-40ca-9c28-4e2931eb61b3" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:24.026955 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.026883    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b034f8-d52d-40ca-9c28-4e2931eb61b3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "47b034f8-d52d-40ca-9c28-4e2931eb61b3" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:24.027004 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.026950    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-kube-api-access-knkm8" (OuterVolumeSpecName: "kube-api-access-knkm8") pod "47b034f8-d52d-40ca-9c28-4e2931eb61b3" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3"). InnerVolumeSpecName "kube-api-access-knkm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:27:24.033347 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.033326    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b034f8-d52d-40ca-9c28-4e2931eb61b3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "47b034f8-d52d-40ca-9c28-4e2931eb61b3" (UID: "47b034f8-d52d-40ca-9c28-4e2931eb61b3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:27:24.124639 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.124601    2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-tls\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:24.124639 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.124632    2552 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/47b034f8-d52d-40ca-9c28-4e2931eb61b3-image-registry-private-configuration\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:24.124639 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.124644    2552 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/47b034f8-d52d-40ca-9c28-4e2931eb61b3-registry-certificates\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:24.124866 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.124652    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47b034f8-d52d-40ca-9c28-4e2931eb61b3-trusted-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:24.124866 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.124661    2552 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/47b034f8-d52d-40ca-9c28-4e2931eb61b3-installation-pull-secrets\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:24.124866 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.124669    2552 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-bound-sa-token\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:24.124866 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.124679    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knkm8\" (UniqueName: \"kubernetes.io/projected/47b034f8-d52d-40ca-9c28-4e2931eb61b3-kube-api-access-knkm8\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:24.124866 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.124687    2552 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/47b034f8-d52d-40ca-9c28-4e2931eb61b3-ca-trust-extracted\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:24.909017 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.908984    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64f498f4fc-7k7bl"
Sep 10 17:27:24.927235 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.927202    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64f498f4fc-7k7bl"]
Sep 10 17:27:24.932936 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:24.932909    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64f498f4fc-7k7bl"]
Sep 10 17:27:26.802595 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:26.802559    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47b034f8-d52d-40ca-9c28-4e2931eb61b3" path="/var/lib/kubelet/pods/47b034f8-d52d-40ca-9c28-4e2931eb61b3/volumes"
Sep 10 17:27:31.843669 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:31.843631    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:27:31.844198 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:31.843919    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="alertmanager" containerID="cri-o://1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74" gracePeriod=120
Sep 10 17:27:31.844198 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:31.843999    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="kube-rbac-proxy" containerID="cri-o://1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522" gracePeriod=120
Sep 10 17:27:31.844198 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:31.843995    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="kube-rbac-proxy-metric" containerID="cri-o://4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a" gracePeriod=120
Sep 10 17:27:31.844198 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:31.844031    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="config-reloader" containerID="cri-o://84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1" gracePeriod=120
Sep 10 17:27:31.844198 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:31.844057    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="prom-label-proxy" containerID="cri-o://29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0" gracePeriod=120
Sep 10 17:27:31.844198 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:31.844000    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="kube-rbac-proxy-web" containerID="cri-o://3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3" gracePeriod=120
Sep 10 17:27:32.944355 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:32.944326    2552 generic.go:356] "Generic (PLEG): container finished" podID="db3f88aa-1a56-4180-852f-0710838303d4" containerID="29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0" exitCode=0
Sep 10 17:27:32.944355 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:32.944349    2552 generic.go:356] "Generic (PLEG): container finished" podID="db3f88aa-1a56-4180-852f-0710838303d4" containerID="1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522" exitCode=0
Sep 10 17:27:32.944355 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:32.944355    2552 generic.go:356] "Generic (PLEG): container finished" podID="db3f88aa-1a56-4180-852f-0710838303d4" containerID="84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1" exitCode=0
Sep 10 17:27:32.944355 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:32.944361    2552 generic.go:356] "Generic (PLEG): container finished" podID="db3f88aa-1a56-4180-852f-0710838303d4" containerID="1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74" exitCode=0
Sep 10 17:27:32.944839 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:32.944401    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3f88aa-1a56-4180-852f-0710838303d4","Type":"ContainerDied","Data":"29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0"}
Sep 10 17:27:32.944839 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:32.944517    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3f88aa-1a56-4180-852f-0710838303d4","Type":"ContainerDied","Data":"1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522"}
Sep 10 17:27:32.944839 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:32.944565    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3f88aa-1a56-4180-852f-0710838303d4","Type":"ContainerDied","Data":"84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1"}
Sep 10 17:27:32.944839 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:32.944608    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3f88aa-1a56-4180-852f-0710838303d4","Type":"ContainerDied","Data":"1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74"}
Sep 10 17:27:33.096048 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.096020    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:33.206557 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.206466    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy\") pod \"db3f88aa-1a56-4180-852f-0710838303d4\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") "
Sep 10 17:27:33.206557 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.206531    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsd99\" (UniqueName: \"kubernetes.io/projected/db3f88aa-1a56-4180-852f-0710838303d4-kube-api-access-jsd99\") pod \"db3f88aa-1a56-4180-852f-0710838303d4\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") "
Sep 10 17:27:33.206768 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.206572    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db3f88aa-1a56-4180-852f-0710838303d4-config-out\") pod \"db3f88aa-1a56-4180-852f-0710838303d4\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") "
Sep 10 17:27:33.206768 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.206593    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-cluster-tls-config\") pod \"db3f88aa-1a56-4180-852f-0710838303d4\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") "
Sep 10 17:27:33.206768 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.206636    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-web-config\") pod \"db3f88aa-1a56-4180-852f-0710838303d4\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") "
Sep 10 17:27:33.206768 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.206669    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db3f88aa-1a56-4180-852f-0710838303d4-metrics-client-ca\") pod \"db3f88aa-1a56-4180-852f-0710838303d4\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") "
Sep 10 17:27:33.206768 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.206697    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db3f88aa-1a56-4180-852f-0710838303d4-alertmanager-trusted-ca-bundle\") pod \"db3f88aa-1a56-4180-852f-0710838303d4\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") "
Sep 10 17:27:33.206768 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.206720    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db3f88aa-1a56-4180-852f-0710838303d4-tls-assets\") pod \"db3f88aa-1a56-4180-852f-0710838303d4\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") "
Sep 10 17:27:33.206768 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.206762    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-main-tls\") pod \"db3f88aa-1a56-4180-852f-0710838303d4\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") "
Sep 10 17:27:33.207118 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.206799    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"db3f88aa-1a56-4180-852f-0710838303d4\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") "
Sep 10 17:27:33.207118 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.206833    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db3f88aa-1a56-4180-852f-0710838303d4-alertmanager-main-db\") pod \"db3f88aa-1a56-4180-852f-0710838303d4\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") "
Sep 10 17:27:33.207118 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.206858    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"db3f88aa-1a56-4180-852f-0710838303d4\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") "
Sep 10 17:27:33.207118 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.206903    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-config-volume\") pod \"db3f88aa-1a56-4180-852f-0710838303d4\" (UID: \"db3f88aa-1a56-4180-852f-0710838303d4\") "
Sep 10 17:27:33.207118 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.207080    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3f88aa-1a56-4180-852f-0710838303d4-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "db3f88aa-1a56-4180-852f-0710838303d4" (UID: "db3f88aa-1a56-4180-852f-0710838303d4"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:27:33.207365 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.207201    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db3f88aa-1a56-4180-852f-0710838303d4-metrics-client-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:33.207927 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.207882    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3f88aa-1a56-4180-852f-0710838303d4-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "db3f88aa-1a56-4180-852f-0710838303d4" (UID: "db3f88aa-1a56-4180-852f-0710838303d4"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:27:33.208073 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.207998    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3f88aa-1a56-4180-852f-0710838303d4-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "db3f88aa-1a56-4180-852f-0710838303d4" (UID: "db3f88aa-1a56-4180-852f-0710838303d4"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:27:33.209607 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.209568    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "db3f88aa-1a56-4180-852f-0710838303d4" (UID: "db3f88aa-1a56-4180-852f-0710838303d4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:33.209838 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.209814    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3f88aa-1a56-4180-852f-0710838303d4-kube-api-access-jsd99" (OuterVolumeSpecName: "kube-api-access-jsd99") pod "db3f88aa-1a56-4180-852f-0710838303d4" (UID: "db3f88aa-1a56-4180-852f-0710838303d4"). InnerVolumeSpecName "kube-api-access-jsd99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:27:33.209931 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.209911    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3f88aa-1a56-4180-852f-0710838303d4-config-out" (OuterVolumeSpecName: "config-out") pod "db3f88aa-1a56-4180-852f-0710838303d4" (UID: "db3f88aa-1a56-4180-852f-0710838303d4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:27:33.210178 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.210146    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "db3f88aa-1a56-4180-852f-0710838303d4" (UID: "db3f88aa-1a56-4180-852f-0710838303d4"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:33.210178 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.210171    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-config-volume" (OuterVolumeSpecName: "config-volume") pod "db3f88aa-1a56-4180-852f-0710838303d4" (UID: "db3f88aa-1a56-4180-852f-0710838303d4"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:33.211708 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.211674    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "db3f88aa-1a56-4180-852f-0710838303d4" (UID: "db3f88aa-1a56-4180-852f-0710838303d4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:33.211708 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.211687    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3f88aa-1a56-4180-852f-0710838303d4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "db3f88aa-1a56-4180-852f-0710838303d4" (UID: "db3f88aa-1a56-4180-852f-0710838303d4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:27:33.211836 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.211821    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "db3f88aa-1a56-4180-852f-0710838303d4" (UID: "db3f88aa-1a56-4180-852f-0710838303d4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:33.214370 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.214342    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "db3f88aa-1a56-4180-852f-0710838303d4" (UID: "db3f88aa-1a56-4180-852f-0710838303d4"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:33.221107 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.221082    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-web-config" (OuterVolumeSpecName: "web-config") pod "db3f88aa-1a56-4180-852f-0710838303d4" (UID: "db3f88aa-1a56-4180-852f-0710838303d4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:33.308388 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.308349    2552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db3f88aa-1a56-4180-852f-0710838303d4-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:33.308388 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.308382    2552 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db3f88aa-1a56-4180-852f-0710838303d4-tls-assets\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:33.308388 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.308392    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-main-tls\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:33.308609 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.308405    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:33.308609 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.308414    2552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db3f88aa-1a56-4180-852f-0710838303d4-alertmanager-main-db\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:33.308609 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.308423    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:33.308609 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.308432    2552 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-config-volume\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:33.308609 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.308440    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:33.308609 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.308449    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsd99\" (UniqueName: \"kubernetes.io/projected/db3f88aa-1a56-4180-852f-0710838303d4-kube-api-access-jsd99\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:33.308609 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.308457    2552 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db3f88aa-1a56-4180-852f-0710838303d4-config-out\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:33.308609 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.308465    2552 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-cluster-tls-config\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:33.308609 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.308472    2552 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db3f88aa-1a56-4180-852f-0710838303d4-web-config\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:33.951362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.951325    2552 generic.go:356] "Generic (PLEG): container finished" podID="db3f88aa-1a56-4180-852f-0710838303d4" containerID="4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a" exitCode=0
Sep 10 17:27:33.951362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.951353    2552 generic.go:356] "Generic (PLEG): container finished" podID="db3f88aa-1a56-4180-852f-0710838303d4" containerID="3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3" exitCode=0
Sep 10 17:27:33.951774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.951399    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3f88aa-1a56-4180-852f-0710838303d4","Type":"ContainerDied","Data":"4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a"}
Sep 10 17:27:33.951774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.951469    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3f88aa-1a56-4180-852f-0710838303d4","Type":"ContainerDied","Data":"3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3"}
Sep 10 17:27:33.951774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.951405    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:33.951774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.951516    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db3f88aa-1a56-4180-852f-0710838303d4","Type":"ContainerDied","Data":"11e467eb9d56f89ca2cc88631bd3e0d15bee1c8080fff01e2c7ca3a567d7e791"}
Sep 10 17:27:33.951774 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.951550    2552 scope.go:117] "RemoveContainer" containerID="29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0"
Sep 10 17:27:33.959924 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.959902    2552 scope.go:117] "RemoveContainer" containerID="4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a"
Sep 10 17:27:33.967104 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.967087    2552 scope.go:117] "RemoveContainer" containerID="1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522"
Sep 10 17:27:33.974086 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.974064    2552 scope.go:117] "RemoveContainer" containerID="3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3"
Sep 10 17:27:33.976057 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.976036    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:27:33.979681 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.979660    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:27:33.982128 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.982111    2552 scope.go:117] "RemoveContainer" containerID="84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1"
Sep 10 17:27:33.988766 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.988747    2552 scope.go:117] "RemoveContainer" containerID="1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74"
Sep 10 17:27:33.995824 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:33.995806    2552 scope.go:117] "RemoveContainer" containerID="c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690"
Sep 10 17:27:34.003324 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.003303    2552 scope.go:117] "RemoveContainer" containerID="29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0"
Sep 10 17:27:34.003662 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:27:34.003643    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0\": container with ID starting with 29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0 not found: ID does not exist" containerID="29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0"
Sep 10 17:27:34.003722 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.003673    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0"} err="failed to get container status \"29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0\": rpc error: code = NotFound desc = could not find container \"29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0\": container with ID starting with 29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0 not found: ID does not exist"
Sep 10 17:27:34.003722 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.003694    2552 scope.go:117] "RemoveContainer" containerID="4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a"
Sep 10 17:27:34.003985 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:27:34.003965    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a\": container with ID starting with 4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a not found: ID does not exist" containerID="4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a"
Sep 10 17:27:34.004027 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.003990    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a"} err="failed to get container status \"4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a\": rpc error: code = NotFound desc = could not find container \"4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a\": container with ID starting with 4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a not found: ID does not exist"
Sep 10 17:27:34.004027 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.004009    2552 scope.go:117] "RemoveContainer" containerID="1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522"
Sep 10 17:27:34.004294 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:27:34.004268    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522\": container with ID starting with 1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522 not found: ID does not exist" containerID="1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522"
Sep 10 17:27:34.004392 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.004300    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522"} err="failed to get container status \"1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522\": rpc error: code = NotFound desc = could not find container \"1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522\": container with ID starting with 1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522 not found: ID does not exist"
Sep 10 17:27:34.004392 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.004316    2552 scope.go:117] "RemoveContainer" containerID="3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3"
Sep 10 17:27:34.004832 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:27:34.004679    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3\": container with ID starting with 3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3 not found: ID does not exist" containerID="3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3"
Sep 10 17:27:34.004832 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.004723    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3"} err="failed to get container status \"3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3\": rpc error: code = NotFound desc = could not find container \"3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3\": container with ID starting with 3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3 not found: ID does not exist"
Sep 10 17:27:34.004832 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.004745    2552 scope.go:117] "RemoveContainer" containerID="84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1"
Sep 10 17:27:34.005079 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:27:34.005053    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1\": container with ID starting with 84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1 not found: ID does not exist" containerID="84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1"
Sep 10 17:27:34.005149 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.005086    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1"} err="failed to get container status \"84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1\": rpc error: code = NotFound desc = could not find container \"84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1\": container with ID starting with 84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1 not found: ID does not exist"
Sep 10 17:27:34.005149 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.005112    2552 scope.go:117] "RemoveContainer" containerID="1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74"
Sep 10 17:27:34.005382 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:27:34.005364    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74\": container with ID starting with 1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74 not found: ID does not exist" containerID="1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74"
Sep 10 17:27:34.005525 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.005474    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74"} err="failed to get container status \"1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74\": rpc error: code = NotFound desc = could not find container \"1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74\": container with ID starting with 1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74 not found: ID does not exist"
Sep 10 17:27:34.005525 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.005524    2552 scope.go:117] "RemoveContainer" containerID="c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690"
Sep 10 17:27:34.005791 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:27:34.005772    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690\": container with ID starting with c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690 not found: ID does not exist" containerID="c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690"
Sep 10 17:27:34.005858 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.005794    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690"} err="failed to get container status \"c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690\": rpc error: code = NotFound desc = could not find container \"c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690\": container with ID starting with c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690 not found: ID does not exist"
Sep 10 17:27:34.005858 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.005807    2552 scope.go:117] "RemoveContainer" containerID="29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0"
Sep 10 17:27:34.006062 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.006044    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0"} err="failed to get container status \"29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0\": rpc error: code = NotFound desc = could not find container \"29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0\": container with ID starting with 29053cd3dac6a69f16a1b7e5c4d850c9ad857024ef465fdefbb0d485192bbcb0 not found: ID does not exist"
Sep 10 17:27:34.006122 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.006062    2552 scope.go:117] "RemoveContainer" containerID="4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a"
Sep 10 17:27:34.006342 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.006321    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a"} err="failed to get container status \"4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a\": rpc error: code = NotFound desc = could not find container \"4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a\": container with ID starting with 4e67e470aa4cef9ea412c3eaeaf91c92a6c75b145cb3431340e2bdf793a7e46a not found: ID does not exist"
Sep 10 17:27:34.006409 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.006342    2552 scope.go:117] "RemoveContainer" containerID="1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522"
Sep 10 17:27:34.006605 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.006580    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522"} err="failed to get container status \"1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522\": rpc error: code = NotFound desc = could not find container \"1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522\": container with ID starting with 1eb3e369df2d73e9d955e66d60c68e8d1ef531abfb503d2a686d9b9624061522 not found: ID does not exist"
Sep 10 17:27:34.006605 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.006604    2552 scope.go:117] "RemoveContainer" containerID="3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3"
Sep 10 17:27:34.006826 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.006806    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3"} err="failed to get container status \"3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3\": rpc error: code = NotFound desc = could not find container \"3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3\": container with ID starting with 3720c0c50e22f64c6b89147761dcd5a619f8dfb343534c59fee9f1d78a836cc3 not found: ID does not exist"
Sep 10 17:27:34.006826 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.006825    2552 scope.go:117] "RemoveContainer" containerID="84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1"
Sep 10 17:27:34.006936 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.006823    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:27:34.007022 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007005    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1"} err="failed to get container status \"84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1\": rpc error: code = NotFound desc = could not find container \"84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1\": container with ID starting with 84b89773488e70096a5fded737e18c8b86edb4bab653203a4566d4d3459b77d1 not found: ID does not exist"
Sep 10 17:27:34.007022 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007021    2552 scope.go:117] "RemoveContainer" containerID="1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74"
Sep 10 17:27:34.007222 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007206    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74"} err="failed to get container status \"1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74\": rpc error: code = NotFound desc = could not find container \"1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74\": container with ID starting with 1598b0ddcf3475adabc747f32e22841f13dbbc0e819565c937e290e5159fda74 not found: ID does not exist"
Sep 10 17:27:34.007222 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007221    2552 scope.go:117] "RemoveContainer" containerID="c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690"
Sep 10 17:27:34.007338 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007225    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52" containerName="console"
Sep 10 17:27:34.007338 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007241    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52" containerName="console"
Sep 10 17:27:34.007338 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007252    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="kube-rbac-proxy"
Sep 10 17:27:34.007338 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007261    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="kube-rbac-proxy"
Sep 10 17:27:34.007338 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007280    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="prom-label-proxy"
Sep 10 17:27:34.007338 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007291    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="prom-label-proxy"
Sep 10 17:27:34.007338 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007305    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="alertmanager"
Sep 10 17:27:34.007338 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007314    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="alertmanager"
Sep 10 17:27:34.007338 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007329    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="kube-rbac-proxy-metric"
Sep 10 17:27:34.007338 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007337    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="kube-rbac-proxy-metric"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007363    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b034f8-d52d-40ca-9c28-4e2931eb61b3" containerName="registry"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007371    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b034f8-d52d-40ca-9c28-4e2931eb61b3" containerName="registry"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007384    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="init-config-reloader"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007392    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="init-config-reloader"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007402    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="config-reloader"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007406    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690"} err="failed to get container status \"c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690\": rpc error: code = NotFound desc = could not find container \"c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690\": container with ID starting with c02fde335c95244d9e67695640cd536db529fb9196a11bfcb6ef12ffe6c5c690 not found: ID does not exist"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007411    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="config-reloader"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007431    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="kube-rbac-proxy-web"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007440    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="kube-rbac-proxy-web"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007544    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="kube-rbac-proxy"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007557    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="alertmanager"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007566    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="kube-rbac-proxy-web"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007578    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b867cf2-06f8-4e7b-a903-b1c2f3ff7d52" containerName="console"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007590    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="config-reloader"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007598    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b034f8-d52d-40ca-9c28-4e2931eb61b3" containerName="registry"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007606    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="prom-label-proxy"
Sep 10 17:27:34.007878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.007614    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="db3f88aa-1a56-4180-852f-0710838303d4" containerName="kube-rbac-proxy-metric"
Sep 10 17:27:34.011574 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.011557    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.014145 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.014084    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 17:27:34.014414 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.014393    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 17:27:34.014487 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.014473    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 17:27:34.014907 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.014598    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 17:27:34.014907 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.014615    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 17:27:34.014907 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.014642    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 17:27:34.014907 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.014653    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 17:27:34.014907 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.014642    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 17:27:34.014907 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.014720    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-d95xn\""
Sep 10 17:27:34.028778 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.028754    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 17:27:34.031879 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.031850    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:27:34.114374 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.114337    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/98993b00-3035-42d4-b263-7dc9aaeaa839-tls-assets\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.114374 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.114377    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.114603 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.114401    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/98993b00-3035-42d4-b263-7dc9aaeaa839-config-out\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.114603 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.114446    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.114603 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.114480    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-web-config\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.114603 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.114526    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98993b00-3035-42d4-b263-7dc9aaeaa839-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.114603 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.114550    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2zxj\" (UniqueName: \"kubernetes.io/projected/98993b00-3035-42d4-b263-7dc9aaeaa839-kube-api-access-p2zxj\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.114603 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.114587    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.114785 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.114609    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/98993b00-3035-42d4-b263-7dc9aaeaa839-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.114785 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.114644    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-config-volume\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.114785 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.114666    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98993b00-3035-42d4-b263-7dc9aaeaa839-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.114785 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.114687    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.114785 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.114758    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.216048 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.215950    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.216048 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.215994    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/98993b00-3035-42d4-b263-7dc9aaeaa839-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.216048 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.216021    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-config-volume\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.216048 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.216043    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98993b00-3035-42d4-b263-7dc9aaeaa839-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.216349 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.216063    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.216412 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.216363    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.216464 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.216413    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/98993b00-3035-42d4-b263-7dc9aaeaa839-tls-assets\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.216464 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.216416    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/98993b00-3035-42d4-b263-7dc9aaeaa839-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.216624 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.216457    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.216624 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.216488    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/98993b00-3035-42d4-b263-7dc9aaeaa839-config-out\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.216624 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.216556    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.216624 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.216602    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-web-config\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.216624 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.216622    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98993b00-3035-42d4-b263-7dc9aaeaa839-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.216864 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.216645    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2zxj\" (UniqueName: \"kubernetes.io/projected/98993b00-3035-42d4-b263-7dc9aaeaa839-kube-api-access-p2zxj\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.217105 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.217033    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98993b00-3035-42d4-b263-7dc9aaeaa839-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.217675 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.217627    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98993b00-3035-42d4-b263-7dc9aaeaa839-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.219347 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.219295    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/98993b00-3035-42d4-b263-7dc9aaeaa839-tls-assets\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.219347 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.219331    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.219803 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.219777    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.219880 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.219791    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-config-volume\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.219880 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.219850    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.219953 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.219926    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.220081 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.220058    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-web-config\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.220551 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.220534    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/98993b00-3035-42d4-b263-7dc9aaeaa839-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.221454 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.221439    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/98993b00-3035-42d4-b263-7dc9aaeaa839-config-out\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.225587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.225560    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2zxj\" (UniqueName: \"kubernetes.io/projected/98993b00-3035-42d4-b263-7dc9aaeaa839-kube-api-access-p2zxj\") pod \"alertmanager-main-0\" (UID: \"98993b00-3035-42d4-b263-7dc9aaeaa839\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.321462 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.321421    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:27:34.598522 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.598458    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:27:34.600529 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:27:34.600502    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98993b00_3035_42d4_b263_7dc9aaeaa839.slice/crio-ff461d24d3f58cf9b5ecb547e3dafb9884e89c5626aa647042bcde91b38e0cc9 WatchSource:0}: Error finding container ff461d24d3f58cf9b5ecb547e3dafb9884e89c5626aa647042bcde91b38e0cc9: Status 404 returned error can't find the container with id ff461d24d3f58cf9b5ecb547e3dafb9884e89c5626aa647042bcde91b38e0cc9
Sep 10 17:27:34.802579 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.802548    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db3f88aa-1a56-4180-852f-0710838303d4" path="/var/lib/kubelet/pods/db3f88aa-1a56-4180-852f-0710838303d4/volumes"
Sep 10 17:27:34.957379 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.957293    2552 generic.go:356] "Generic (PLEG): container finished" podID="98993b00-3035-42d4-b263-7dc9aaeaa839" containerID="6fe5448eadb62588998b6b30f5a7e26e7e7f90660871d1c0de1d3e2e9255f3bb" exitCode=0
Sep 10 17:27:34.957379 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.957340    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98993b00-3035-42d4-b263-7dc9aaeaa839","Type":"ContainerDied","Data":"6fe5448eadb62588998b6b30f5a7e26e7e7f90660871d1c0de1d3e2e9255f3bb"}
Sep 10 17:27:34.957798 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:34.957397    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98993b00-3035-42d4-b263-7dc9aaeaa839","Type":"ContainerStarted","Data":"ff461d24d3f58cf9b5ecb547e3dafb9884e89c5626aa647042bcde91b38e0cc9"}
Sep 10 17:27:35.965150 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:35.965115    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98993b00-3035-42d4-b263-7dc9aaeaa839","Type":"ContainerStarted","Data":"ce92fe4997b246fd15b3286de236a8aed215d0b83183370b4141c65141e360d3"}
Sep 10 17:27:35.965622 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:35.965192    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98993b00-3035-42d4-b263-7dc9aaeaa839","Type":"ContainerStarted","Data":"08ac90d9d5c33c049f96351b6994eb57e7e819d0736bee6f11c24a5aca720ec9"}
Sep 10 17:27:35.965622 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:35.965240    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98993b00-3035-42d4-b263-7dc9aaeaa839","Type":"ContainerStarted","Data":"ddbd251990d2d3dfdeb6bc61f0c1398ebc6a0a08593c20be2254e39f9964a88e"}
Sep 10 17:27:35.965622 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:35.965284    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98993b00-3035-42d4-b263-7dc9aaeaa839","Type":"ContainerStarted","Data":"232b4228bb6f933ad8493073b409b82441a17f0c40c4e10b6863c9d913e432b3"}
Sep 10 17:27:35.965622 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:35.965325    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98993b00-3035-42d4-b263-7dc9aaeaa839","Type":"ContainerStarted","Data":"836f4d492a981347bb18c34cdb5fe9abca2f65c8f5ccb3b954d457047e6b6652"}
Sep 10 17:27:35.965622 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:35.965364    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98993b00-3035-42d4-b263-7dc9aaeaa839","Type":"ContainerStarted","Data":"683d6e60e8bcbd3bdbb84a66f3cfc04d9195a21a2af8c79206a89404520fa6be"}
Sep 10 17:27:36.000082 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.000021    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.987431563 podStartE2EDuration="3.000000968s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.569405ms" isStatefulPod=false podCreationTimestamp="2026-09-10 17:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:27:35.997847048 +0000 UTC m=+227.756374119" watchObservedRunningTime="2026-09-10 17:27:36.000000968 +0000 UTC m=+227.758528106"
Sep 10 17:27:36.096187 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.096153    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:27:36.096487 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.096452    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="prometheus" containerID="cri-o://6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530" gracePeriod=600
Sep 10 17:27:36.096487 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.096478    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="thanos-sidecar" containerID="cri-o://3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f" gracePeriod=600
Sep 10 17:27:36.096753 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.096535    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="kube-rbac-proxy-web" containerID="cri-o://097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3" gracePeriod=600
Sep 10 17:27:36.096753 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.096471    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="kube-rbac-proxy" containerID="cri-o://506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538" gracePeriod=600
Sep 10 17:27:36.096753 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.096554    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5" gracePeriod=600
Sep 10 17:27:36.096753 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.096542    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="config-reloader" containerID="cri-o://649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644" gracePeriod=600
Sep 10 17:27:36.327032 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.327006    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:36.435099 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435067    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-0\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.435261 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435109    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-metrics-client-ca\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.435261 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435139    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d905844a-59bb-494a-bd3e-1f1eee385869-config-out\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.435261 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435168    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-2\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.435261 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435212    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-metrics-client-certs\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.435261 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435233    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-tls\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.435567 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435273    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-serving-certs-ca-bundle\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.435567 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435300    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-trusted-ca-bundle\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.435567 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435335    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-grpc-tls\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.435567 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435364    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-1\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.435567 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435403    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbd5x\" (UniqueName: \"kubernetes.io/projected/d905844a-59bb-494a-bd3e-1f1eee385869-kube-api-access-pbd5x\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.435567 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435437    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.435567 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435465    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.435567 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435514    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-kube-rbac-proxy\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.435567 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435559    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-web-config\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.436008 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435592    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-db\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.436008 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435606    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:27:36.436008 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435641    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d905844a-59bb-494a-bd3e-1f1eee385869-tls-assets\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.436008 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435670    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-kubelet-serving-ca-bundle\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.436008 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435705    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-config\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.436008 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435737    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-thanos-prometheus-http-client-file\") pod \"d905844a-59bb-494a-bd3e-1f1eee385869\" (UID: \"d905844a-59bb-494a-bd3e-1f1eee385869\") "
Sep 10 17:27:36.436008 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.435823    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:27:36.436355 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.436023    2552 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-metrics-client-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.436355 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.436044    2552 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.436466 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.436386    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:27:36.437377 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.436739    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:27:36.437377 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.437152    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:27:36.437785 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.437757    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:27:36.438075 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.438044    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:36.438075 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.438057    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:36.438336 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.438300    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:36.438437 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.438333    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:36.438437 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.438371    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:27:36.438592 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.438566    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:36.438855 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.438830    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:27:36.438946 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.438906    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d905844a-59bb-494a-bd3e-1f1eee385869-config-out" (OuterVolumeSpecName: "config-out") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:27:36.439632 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.439600    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:36.440340 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.440317    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:36.440555 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.440528    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-config" (OuterVolumeSpecName: "config") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:36.440625 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.440552    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d905844a-59bb-494a-bd3e-1f1eee385869-kube-api-access-pbd5x" (OuterVolumeSpecName: "kube-api-access-pbd5x") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "kube-api-access-pbd5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:27:36.440625 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.440613    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d905844a-59bb-494a-bd3e-1f1eee385869-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:27:36.449755 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.449722    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-web-config" (OuterVolumeSpecName: "web-config") pod "d905844a-59bb-494a-bd3e-1f1eee385869" (UID: "d905844a-59bb-494a-bd3e-1f1eee385869"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:36.536980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.536891    2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.536980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.536921    2552 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-grpc-tls\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.536980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.536932    2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.536980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.536941    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbd5x\" (UniqueName: \"kubernetes.io/projected/d905844a-59bb-494a-bd3e-1f1eee385869-kube-api-access-pbd5x\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.536980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.536950    2552 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.536980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.536960    2552 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.536980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.536968    2552 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-kube-rbac-proxy\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.536980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.536977    2552 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-web-config\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.536980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.536985    2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-db\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.537352 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.536992    2552 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d905844a-59bb-494a-bd3e-1f1eee385869-tls-assets\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.537352 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.537002    2552 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.537352 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.537010    2552 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-config\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.537352 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.537018    2552 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.537352 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.537028    2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.537352 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.537042    2552 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d905844a-59bb-494a-bd3e-1f1eee385869-config-out\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.537352 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.537052    2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d905844a-59bb-494a-bd3e-1f1eee385869-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.537352 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.537062    2552 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-metrics-client-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.537352 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.537070    2552 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d905844a-59bb-494a-bd3e-1f1eee385869-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:27:36.972884 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.972853    2552 generic.go:356] "Generic (PLEG): container finished" podID="d905844a-59bb-494a-bd3e-1f1eee385869" containerID="b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5" exitCode=0
Sep 10 17:27:36.972884 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.972878    2552 generic.go:356] "Generic (PLEG): container finished" podID="d905844a-59bb-494a-bd3e-1f1eee385869" containerID="506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538" exitCode=0
Sep 10 17:27:36.972884 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.972886    2552 generic.go:356] "Generic (PLEG): container finished" podID="d905844a-59bb-494a-bd3e-1f1eee385869" containerID="097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3" exitCode=0
Sep 10 17:27:36.972884 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.972891    2552 generic.go:356] "Generic (PLEG): container finished" podID="d905844a-59bb-494a-bd3e-1f1eee385869" containerID="3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f" exitCode=0
Sep 10 17:27:36.972884 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.972896    2552 generic.go:356] "Generic (PLEG): container finished" podID="d905844a-59bb-494a-bd3e-1f1eee385869" containerID="649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644" exitCode=0
Sep 10 17:27:36.973470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.972901    2552 generic.go:356] "Generic (PLEG): container finished" podID="d905844a-59bb-494a-bd3e-1f1eee385869" containerID="6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530" exitCode=0
Sep 10 17:27:36.973470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.972925    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:36.973470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.972937    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerDied","Data":"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"}
Sep 10 17:27:36.973470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.973011    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerDied","Data":"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"}
Sep 10 17:27:36.973470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.973049    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerDied","Data":"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"}
Sep 10 17:27:36.973470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.973079    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerDied","Data":"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"}
Sep 10 17:27:36.973470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.973110    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerDied","Data":"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"}
Sep 10 17:27:36.973470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.973135    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerDied","Data":"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"}
Sep 10 17:27:36.973470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.973080    2552 scope.go:117] "RemoveContainer" containerID="b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"
Sep 10 17:27:36.973470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.973158    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d905844a-59bb-494a-bd3e-1f1eee385869","Type":"ContainerDied","Data":"cd2d13131fea88a2341530f85c2aa555bb41e3f494cdd8c7ed3d8d76db03d98b"}
Sep 10 17:27:36.981573 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.981550    2552 scope.go:117] "RemoveContainer" containerID="506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"
Sep 10 17:27:36.988602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.988487    2552 scope.go:117] "RemoveContainer" containerID="097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"
Sep 10 17:27:36.995820 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.995802    2552 scope.go:117] "RemoveContainer" containerID="3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"
Sep 10 17:27:36.996215 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:36.996188    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:27:37.000656 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.000632    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:27:37.003283 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.003266    2552 scope.go:117] "RemoveContainer" containerID="649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"
Sep 10 17:27:37.010054 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.010036    2552 scope.go:117] "RemoveContainer" containerID="6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"
Sep 10 17:27:37.017622 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.017591    2552 scope.go:117] "RemoveContainer" containerID="97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"
Sep 10 17:27:37.024500 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.024469    2552 scope.go:117] "RemoveContainer" containerID="b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"
Sep 10 17:27:37.024789 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:27:37.024768    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5\": container with ID starting with b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5 not found: ID does not exist" containerID="b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"
Sep 10 17:27:37.024865 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.024801    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"} err="failed to get container status \"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5\": rpc error: code = NotFound desc = could not find container \"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5\": container with ID starting with b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5 not found: ID does not exist"
Sep 10 17:27:37.024865 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.024826    2552 scope.go:117] "RemoveContainer" containerID="506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"
Sep 10 17:27:37.025091 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:27:37.025073    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538\": container with ID starting with 506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538 not found: ID does not exist" containerID="506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"
Sep 10 17:27:37.025132 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.025100    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"} err="failed to get container status \"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538\": rpc error: code = NotFound desc = could not find container \"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538\": container with ID starting with 506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538 not found: ID does not exist"
Sep 10 17:27:37.025132 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.025116    2552 scope.go:117] "RemoveContainer" containerID="097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"
Sep 10 17:27:37.025318 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:27:37.025302    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3\": container with ID starting with 097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3 not found: ID does not exist" containerID="097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"
Sep 10 17:27:37.025363 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.025322    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"} err="failed to get container status \"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3\": rpc error: code = NotFound desc = could not find container \"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3\": container with ID starting with 097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3 not found: ID does not exist"
Sep 10 17:27:37.025363 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.025333    2552 scope.go:117] "RemoveContainer" containerID="3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"
Sep 10 17:27:37.025611 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:27:37.025589    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f\": container with ID starting with 3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f not found: ID does not exist" containerID="3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"
Sep 10 17:27:37.025656 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.025624    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"} err="failed to get container status \"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f\": rpc error: code = NotFound desc = could not find container \"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f\": container with ID starting with 3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f not found: ID does not exist"
Sep 10 17:27:37.025656 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.025646    2552 scope.go:117] "RemoveContainer" containerID="649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"
Sep 10 17:27:37.025930 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:27:37.025912    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644\": container with ID starting with 649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644 not found: ID does not exist" containerID="649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"
Sep 10 17:27:37.025967 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.025935    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"} err="failed to get container status \"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644\": rpc error: code = NotFound desc = could not find container \"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644\": container with ID starting with 649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644 not found: ID does not exist"
Sep 10 17:27:37.025967 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.025952    2552 scope.go:117] "RemoveContainer" containerID="6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"
Sep 10 17:27:37.026166 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:27:37.026152    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530\": container with ID starting with 6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530 not found: ID does not exist" containerID="6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"
Sep 10 17:27:37.026206 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.026168    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"} err="failed to get container status \"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530\": rpc error: code = NotFound desc = could not find container \"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530\": container with ID starting with 6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530 not found: ID does not exist"
Sep 10 17:27:37.026206 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.026178    2552 scope.go:117] "RemoveContainer" containerID="97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"
Sep 10 17:27:37.026397 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:27:37.026380    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0\": container with ID starting with 97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0 not found: ID does not exist" containerID="97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"
Sep 10 17:27:37.026434 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.026402    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"} err="failed to get container status \"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0\": rpc error: code = NotFound desc = could not find container \"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0\": container with ID starting with 97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0 not found: ID does not exist"
Sep 10 17:27:37.026434 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.026417    2552 scope.go:117] "RemoveContainer" containerID="b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"
Sep 10 17:27:37.026710 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.026692    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"} err="failed to get container status \"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5\": rpc error: code = NotFound desc = could not find container \"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5\": container with ID starting with b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5 not found: ID does not exist"
Sep 10 17:27:37.026778 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.026710    2552 scope.go:117] "RemoveContainer" containerID="506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"
Sep 10 17:27:37.026992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.026960    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"} err="failed to get container status \"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538\": rpc error: code = NotFound desc = could not find container \"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538\": container with ID starting with 506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538 not found: ID does not exist"
Sep 10 17:27:37.026992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.026985    2552 scope.go:117] "RemoveContainer" containerID="097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"
Sep 10 17:27:37.027329 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.027285    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"} err="failed to get container status \"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3\": rpc error: code = NotFound desc = could not find container \"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3\": container with ID starting with 097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3 not found: ID does not exist"
Sep 10 17:27:37.027329 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.027303    2552 scope.go:117] "RemoveContainer" containerID="3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"
Sep 10 17:27:37.027607 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.027586    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"} err="failed to get container status \"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f\": rpc error: code = NotFound desc = could not find container \"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f\": container with ID starting with 3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f not found: ID does not exist"
Sep 10 17:27:37.027680 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.027608    2552 scope.go:117] "RemoveContainer" containerID="649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.028189    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"} err="failed to get container status \"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644\": rpc error: code = NotFound desc = could not find container \"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644\": container with ID starting with 649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644 not found: ID does not exist"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.028214    2552 scope.go:117] "RemoveContainer" containerID="6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.028523    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"} err="failed to get container status \"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530\": rpc error: code = NotFound desc = could not find container \"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530\": container with ID starting with 6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530 not found: ID does not exist"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.028545    2552 scope.go:117] "RemoveContainer" containerID="97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030075    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030462    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="thanos-sidecar"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030477    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="thanos-sidecar"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030506    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="kube-rbac-proxy"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030516    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="kube-rbac-proxy"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030529    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="init-config-reloader"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030537    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="init-config-reloader"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030555    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="kube-rbac-proxy-thanos"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030562    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="kube-rbac-proxy-thanos"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030573    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="prometheus"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030582    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="prometheus"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030603    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="kube-rbac-proxy-web"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030611    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="kube-rbac-proxy-web"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030624    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="config-reloader"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030631    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="config-reloader"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030702    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="thanos-sidecar"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030713    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="kube-rbac-proxy"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030722    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="prometheus"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030731    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="kube-rbac-proxy-web"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030739    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="kube-rbac-proxy-thanos"
Sep 10 17:27:37.030800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030749    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" containerName="config-reloader"
Sep 10 17:27:37.031803 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030858    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"} err="failed to get container status \"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0\": rpc error: code = NotFound desc = could not find container \"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0\": container with ID starting with 97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0 not found: ID does not exist"
Sep 10 17:27:37.031803 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.030883    2552 scope.go:117] "RemoveContainer" containerID="b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"
Sep 10 17:27:37.031803 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.031451    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"} err="failed to get container status \"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5\": rpc error: code = NotFound desc = could not find container \"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5\": container with ID starting with b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5 not found: ID does not exist"
Sep 10 17:27:37.031803 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.031469    2552 scope.go:117] "RemoveContainer" containerID="506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"
Sep 10 17:27:37.031803 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.031767    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"} err="failed to get container status \"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538\": rpc error: code = NotFound desc = could not find container \"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538\": container with ID starting with 506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538 not found: ID does not exist"
Sep 10 17:27:37.031803 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.031788    2552 scope.go:117] "RemoveContainer" containerID="097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"
Sep 10 17:27:37.032047 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.032027    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"} err="failed to get container status \"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3\": rpc error: code = NotFound desc = could not find container \"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3\": container with ID starting with 097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3 not found: ID does not exist"
Sep 10 17:27:37.032047 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.032046    2552 scope.go:117] "RemoveContainer" containerID="3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"
Sep 10 17:27:37.032274 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.032254    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"} err="failed to get container status \"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f\": rpc error: code = NotFound desc = could not find container \"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f\": container with ID starting with 3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f not found: ID does not exist"
Sep 10 17:27:37.032330 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.032274    2552 scope.go:117] "RemoveContainer" containerID="649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"
Sep 10 17:27:37.032517 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.032474    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"} err="failed to get container status \"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644\": rpc error: code = NotFound desc = could not find container \"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644\": container with ID starting with 649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644 not found: ID does not exist"
Sep 10 17:27:37.032517 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.032515    2552 scope.go:117] "RemoveContainer" containerID="6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"
Sep 10 17:27:37.032772 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.032754    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"} err="failed to get container status \"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530\": rpc error: code = NotFound desc = could not find container \"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530\": container with ID starting with 6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530 not found: ID does not exist"
Sep 10 17:27:37.032842 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.032775    2552 scope.go:117] "RemoveContainer" containerID="97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"
Sep 10 17:27:37.033024 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.033003    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"} err="failed to get container status \"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0\": rpc error: code = NotFound desc = could not find container \"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0\": container with ID starting with 97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0 not found: ID does not exist"
Sep 10 17:27:37.033024 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.033022    2552 scope.go:117] "RemoveContainer" containerID="b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"
Sep 10 17:27:37.033261 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.033243    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"} err="failed to get container status \"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5\": rpc error: code = NotFound desc = could not find container \"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5\": container with ID starting with b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5 not found: ID does not exist"
Sep 10 17:27:37.033261 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.033261    2552 scope.go:117] "RemoveContainer" containerID="506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"
Sep 10 17:27:37.033476 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.033458    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"} err="failed to get container status \"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538\": rpc error: code = NotFound desc = could not find container \"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538\": container with ID starting with 506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538 not found: ID does not exist"
Sep 10 17:27:37.033539 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.033477    2552 scope.go:117] "RemoveContainer" containerID="097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"
Sep 10 17:27:37.033715 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.033697    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"} err="failed to get container status \"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3\": rpc error: code = NotFound desc = could not find container \"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3\": container with ID starting with 097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3 not found: ID does not exist"
Sep 10 17:27:37.033715 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.033715    2552 scope.go:117] "RemoveContainer" containerID="3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"
Sep 10 17:27:37.033906 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.033891    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"} err="failed to get container status \"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f\": rpc error: code = NotFound desc = could not find container \"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f\": container with ID starting with 3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f not found: ID does not exist"
Sep 10 17:27:37.033959 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.033908    2552 scope.go:117] "RemoveContainer" containerID="649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"
Sep 10 17:27:37.034139 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.034122    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"} err="failed to get container status \"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644\": rpc error: code = NotFound desc = could not find container \"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644\": container with ID starting with 649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644 not found: ID does not exist"
Sep 10 17:27:37.034186 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.034140    2552 scope.go:117] "RemoveContainer" containerID="6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"
Sep 10 17:27:37.034587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.034565    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"} err="failed to get container status \"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530\": rpc error: code = NotFound desc = could not find container \"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530\": container with ID starting with 6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530 not found: ID does not exist"
Sep 10 17:27:37.034587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.034585    2552 scope.go:117] "RemoveContainer" containerID="97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"
Sep 10 17:27:37.034790 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.034773    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"} err="failed to get container status \"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0\": rpc error: code = NotFound desc = could not find container \"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0\": container with ID starting with 97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0 not found: ID does not exist"
Sep 10 17:27:37.034854 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.034789    2552 scope.go:117] "RemoveContainer" containerID="b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"
Sep 10 17:27:37.035047 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.035029    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"} err="failed to get container status \"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5\": rpc error: code = NotFound desc = could not find container \"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5\": container with ID starting with b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5 not found: ID does not exist"
Sep 10 17:27:37.035085 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.035049    2552 scope.go:117] "RemoveContainer" containerID="506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"
Sep 10 17:27:37.035085 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.035063    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.035456 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.035319    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"} err="failed to get container status \"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538\": rpc error: code = NotFound desc = could not find container \"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538\": container with ID starting with 506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538 not found: ID does not exist"
Sep 10 17:27:37.035456 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.035350    2552 scope.go:117] "RemoveContainer" containerID="097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"
Sep 10 17:27:37.035729 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.035695    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"} err="failed to get container status \"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3\": rpc error: code = NotFound desc = could not find container \"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3\": container with ID starting with 097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3 not found: ID does not exist"
Sep 10 17:27:37.035729 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.035727    2552 scope.go:117] "RemoveContainer" containerID="3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"
Sep 10 17:27:37.035996 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.035979    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"} err="failed to get container status \"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f\": rpc error: code = NotFound desc = could not find container \"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f\": container with ID starting with 3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f not found: ID does not exist"
Sep 10 17:27:37.035996 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.035996    2552 scope.go:117] "RemoveContainer" containerID="649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"
Sep 10 17:27:37.036230 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.036212    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"} err="failed to get container status \"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644\": rpc error: code = NotFound desc = could not find container \"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644\": container with ID starting with 649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644 not found: ID does not exist"
Sep 10 17:27:37.036284 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.036229    2552 scope.go:117] "RemoveContainer" containerID="6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"
Sep 10 17:27:37.036479 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.036456    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"} err="failed to get container status \"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530\": rpc error: code = NotFound desc = could not find container \"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530\": container with ID starting with 6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530 not found: ID does not exist"
Sep 10 17:27:37.036550 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.036482    2552 scope.go:117] "RemoveContainer" containerID="97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"
Sep 10 17:27:37.036805 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.036784    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"} err="failed to get container status \"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0\": rpc error: code = NotFound desc = could not find container \"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0\": container with ID starting with 97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0 not found: ID does not exist"
Sep 10 17:27:37.036878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.036806    2552 scope.go:117] "RemoveContainer" containerID="b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"
Sep 10 17:27:37.037061 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.037045    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5"} err="failed to get container status \"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5\": rpc error: code = NotFound desc = could not find container \"b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5\": container with ID starting with b7457427b2349647cc51ed29113c493531558b897224bd6e0d1287e3585031e5 not found: ID does not exist"
Sep 10 17:27:37.037105 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.037062    2552 scope.go:117] "RemoveContainer" containerID="506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"
Sep 10 17:27:37.037308 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.037291    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538"} err="failed to get container status \"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538\": rpc error: code = NotFound desc = could not find container \"506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538\": container with ID starting with 506cc0f41b989de02d5e51a940e889285f748390ec63939f9f7457fdeb701538 not found: ID does not exist"
Sep 10 17:27:37.037353 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.037311    2552 scope.go:117] "RemoveContainer" containerID="097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"
Sep 10 17:27:37.037670 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.037636    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3"} err="failed to get container status \"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3\": rpc error: code = NotFound desc = could not find container \"097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3\": container with ID starting with 097621af9acfe6de29afca49c0a0df0fa0b15692e14fc654bc3ef6c2be27edc3 not found: ID does not exist"
Sep 10 17:27:37.037670 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.037665    2552 scope.go:117] "RemoveContainer" containerID="3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"
Sep 10 17:27:37.037807 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.037784    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 17:27:37.038017 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.037803    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 17:27:37.038017 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.037788    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 17:27:37.038017 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.037788    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 17:27:37.038017 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.037843    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 17:27:37.038017 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.037911    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5asi0i0iho0uo\""
Sep 10 17:27:37.038317 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.038044    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f"} err="failed to get container status \"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f\": rpc error: code = NotFound desc = could not find container \"3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f\": container with ID starting with 3cc81af95d99c0f8e06824a2ecfbffb52a69f8cadc3f700a1c37d4e1bb45585f not found: ID does not exist"
Sep 10 17:27:37.038317 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.038066    2552 scope.go:117] "RemoveContainer" containerID="649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"
Sep 10 17:27:37.038317 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.038188    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 17:27:37.038317 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.038196    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 17:27:37.038317 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.038297    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 17:27:37.038570 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.038324    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644"} err="failed to get container status \"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644\": rpc error: code = NotFound desc = could not find container \"649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644\": container with ID starting with 649b5de558e9aa9a388752de45b72c814fff920d843846363031744431fbe644 not found: ID does not exist"
Sep 10 17:27:37.038570 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.038346    2552 scope.go:117] "RemoveContainer" containerID="6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"
Sep 10 17:27:37.038570 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.038395    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 17:27:37.038570 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.038545    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-srxdj\""
Sep 10 17:27:37.038775 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.038685    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530"} err="failed to get container status \"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530\": rpc error: code = NotFound desc = could not find container \"6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530\": container with ID starting with 6dd92cc137b65c2d11f82ae03e54a63a42c16b27bfe2034fda91dc3299f1a530 not found: ID does not exist"
Sep 10 17:27:37.038775 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.038704    2552 scope.go:117] "RemoveContainer" containerID="97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"
Sep 10 17:27:37.038775 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.038743    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 17:27:37.039041 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.039017    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0"} err="failed to get container status \"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0\": rpc error: code = NotFound desc = could not find container \"97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0\": container with ID starting with 97c50c64f4a5265bf9f972cf4c57f1bc1ca8152c65bedc86ce34b74aa5f52da0 not found: ID does not exist"
Sep 10 17:27:37.039141 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.039116    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 17:27:37.039202 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.039148    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 17:27:37.040950 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.040930    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 17:27:37.045128 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.045108    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 17:27:37.054751 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.054729    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:27:37.143580 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.143540    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.143770 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.143587    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-web-config\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.143770 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.143613    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.143770 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.143640    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.143770 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.143717    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.143911 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.143847    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.143946 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.143932    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.143983 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.143954    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/68bb510b-a2db-49b9-a21b-a9ab2007ab85-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.144015 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.143992    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68bb510b-a2db-49b9-a21b-a9ab2007ab85-config-out\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.144045 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.144023    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.144075 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.144058    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.144126 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.144112    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.144161 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.144133    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.144201 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.144166    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.144250 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.144228    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.144331 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.144311    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-config\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.144375 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.144354    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.144409 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.144395    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xb8c\" (UniqueName: \"kubernetes.io/projected/68bb510b-a2db-49b9-a21b-a9ab2007ab85-kube-api-access-9xb8c\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.144455 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.144443    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68bb510b-a2db-49b9-a21b-a9ab2007ab85-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.144555 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.144521    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.245862 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.245810    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.245862 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.245863    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246072 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.245888    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246072 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.245912    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246072 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.245935    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-config\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246072 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.245949    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246072 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.245976    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xb8c\" (UniqueName: \"kubernetes.io/projected/68bb510b-a2db-49b9-a21b-a9ab2007ab85-kube-api-access-9xb8c\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246072 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.246001    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68bb510b-a2db-49b9-a21b-a9ab2007ab85-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246339 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.246023    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246339 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.246140    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246339 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.246172    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-web-config\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246339 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.246197    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246339 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.246229    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246339 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.246261    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246339 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.246292    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246339 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.246335    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246747 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.246361    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/68bb510b-a2db-49b9-a21b-a9ab2007ab85-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246747 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.246390    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68bb510b-a2db-49b9-a21b-a9ab2007ab85-config-out\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246747 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.246411    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246747 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.246441    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246747 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.246585    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.246747 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.246673    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.247585 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.247555    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.249382 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.249099    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.249382 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.249170    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.249382 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.249192    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-config\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.249382 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.249274    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.249693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.249407    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.249693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.249606    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/68bb510b-a2db-49b9-a21b-a9ab2007ab85-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.249693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.249640    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.249867 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.249845    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.249918 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.249885    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.250408 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.250366    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.250593 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.250570    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-web-config\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.256636 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.251368    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68bb510b-a2db-49b9-a21b-a9ab2007ab85-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.256636 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.253742    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/68bb510b-a2db-49b9-a21b-a9ab2007ab85-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.256636 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.253813    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68bb510b-a2db-49b9-a21b-a9ab2007ab85-config-out\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.256636 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.254105    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.257629 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.257608    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68bb510b-a2db-49b9-a21b-a9ab2007ab85-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.259291 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.259272    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xb8c\" (UniqueName: \"kubernetes.io/projected/68bb510b-a2db-49b9-a21b-a9ab2007ab85-kube-api-access-9xb8c\") pod \"prometheus-k8s-0\" (UID: \"68bb510b-a2db-49b9-a21b-a9ab2007ab85\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.346026 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.345976    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:37.629989 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.629963    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:27:37.632291 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:27:37.632267    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bb510b_a2db_49b9_a21b_a9ab2007ab85.slice/crio-d4f2f39e75aaad114e43262c107afb8c68d7b1fd0d4259d798bea5dd5b87e648 WatchSource:0}: Error finding container d4f2f39e75aaad114e43262c107afb8c68d7b1fd0d4259d798bea5dd5b87e648: Status 404 returned error can't find the container with id d4f2f39e75aaad114e43262c107afb8c68d7b1fd0d4259d798bea5dd5b87e648
Sep 10 17:27:37.981904 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.981814    2552 generic.go:356] "Generic (PLEG): container finished" podID="68bb510b-a2db-49b9-a21b-a9ab2007ab85" containerID="075eb8d833e07cc336f3a12043019ea3f202a75e1132f418c60810b107704a2e" exitCode=0
Sep 10 17:27:37.981904 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.981885    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68bb510b-a2db-49b9-a21b-a9ab2007ab85","Type":"ContainerDied","Data":"075eb8d833e07cc336f3a12043019ea3f202a75e1132f418c60810b107704a2e"}
Sep 10 17:27:37.982400 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:37.981960    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68bb510b-a2db-49b9-a21b-a9ab2007ab85","Type":"ContainerStarted","Data":"d4f2f39e75aaad114e43262c107afb8c68d7b1fd0d4259d798bea5dd5b87e648"}
Sep 10 17:27:38.803383 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:38.803350    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d905844a-59bb-494a-bd3e-1f1eee385869" path="/var/lib/kubelet/pods/d905844a-59bb-494a-bd3e-1f1eee385869/volumes"
Sep 10 17:27:38.989287 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:38.989255    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68bb510b-a2db-49b9-a21b-a9ab2007ab85","Type":"ContainerStarted","Data":"4779c0796ea7b28a3b42b2213849fa459b95b990bf684e396c4db3fadd6d4e95"}
Sep 10 17:27:38.989665 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:38.989316    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68bb510b-a2db-49b9-a21b-a9ab2007ab85","Type":"ContainerStarted","Data":"afc776139a861967f4cfa580830ae5600369f47ebe9715af51e594c814dd46a1"}
Sep 10 17:27:38.989665 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:38.989344    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68bb510b-a2db-49b9-a21b-a9ab2007ab85","Type":"ContainerStarted","Data":"db829b8833cf43eab1d699a3ecc58b87e9d088804ec02383b6b312d59acba9d2"}
Sep 10 17:27:38.989665 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:38.989369    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68bb510b-a2db-49b9-a21b-a9ab2007ab85","Type":"ContainerStarted","Data":"b2d059fdc599874abc25e13d99eac0d563aca86be5fdb6c1337923ebde175bae"}
Sep 10 17:27:38.989665 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:38.989392    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68bb510b-a2db-49b9-a21b-a9ab2007ab85","Type":"ContainerStarted","Data":"c91718f37701ad943547e1ab7311a16e442360e8d9190735c9810761f6365bf0"}
Sep 10 17:27:38.989665 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:38.989416    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"68bb510b-a2db-49b9-a21b-a9ab2007ab85","Type":"ContainerStarted","Data":"82d9db3580dc33ae968330c70140a4ec3a939bdf7eac84dfeaf407d091a9460f"}
Sep 10 17:27:39.019506 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:39.019428    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.000069449 podStartE2EDuration="2.01940693s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.337481ms" isStatefulPod=false podCreationTimestamp="2026-09-10 17:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:27:39.017351016 +0000 UTC m=+230.775878106" watchObservedRunningTime="2026-09-10 17:27:39.01940693 +0000 UTC m=+230.777934000"
Sep 10 17:27:42.346301 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:42.346244    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:27:44.735400 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:27:44.735362    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8c8ffbcdb-mmqhc"]
Sep 10 17:28:09.755930 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:09.755870    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8c8ffbcdb-mmqhc" podUID="a38850ba-1f03-4937-8948-07de844bc2eb" containerName="console" containerID="cri-o://1b0837907c008c4627848d0bf4d6a358ab648cfafde00eced96af0512b8d7a6f" gracePeriod=15
Sep 10 17:28:10.001524 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.001486    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:28:10.122176 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.122136    2552 generic.go:356] "Generic (PLEG): container finished" podID="a38850ba-1f03-4937-8948-07de844bc2eb" containerID="1b0837907c008c4627848d0bf4d6a358ab648cfafde00eced96af0512b8d7a6f" exitCode=0
Sep 10 17:28:10.122348 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.122181    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c8ffbcdb-mmqhc"
Sep 10 17:28:10.122348 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.122216    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c8ffbcdb-mmqhc" event={"ID":"a38850ba-1f03-4937-8948-07de844bc2eb","Type":"ContainerDied","Data":"1b0837907c008c4627848d0bf4d6a358ab648cfafde00eced96af0512b8d7a6f"}
Sep 10 17:28:10.122348 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.122285    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c8ffbcdb-mmqhc" event={"ID":"a38850ba-1f03-4937-8948-07de844bc2eb","Type":"ContainerDied","Data":"cebaae6eab2e2c41e0b8d9d77bc9a287f3746809cbf54d7f5c4ae3f034629615"}
Sep 10 17:28:10.122348 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.122305    2552 scope.go:117] "RemoveContainer" containerID="1b0837907c008c4627848d0bf4d6a358ab648cfafde00eced96af0512b8d7a6f"
Sep 10 17:28:10.130757 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.130741    2552 scope.go:117] "RemoveContainer" containerID="1b0837907c008c4627848d0bf4d6a358ab648cfafde00eced96af0512b8d7a6f"
Sep 10 17:28:10.131041 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:28:10.131018    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b0837907c008c4627848d0bf4d6a358ab648cfafde00eced96af0512b8d7a6f\": container with ID starting with 1b0837907c008c4627848d0bf4d6a358ab648cfafde00eced96af0512b8d7a6f not found: ID does not exist" containerID="1b0837907c008c4627848d0bf4d6a358ab648cfafde00eced96af0512b8d7a6f"
Sep 10 17:28:10.131106 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.131050    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b0837907c008c4627848d0bf4d6a358ab648cfafde00eced96af0512b8d7a6f"} err="failed to get container status \"1b0837907c008c4627848d0bf4d6a358ab648cfafde00eced96af0512b8d7a6f\": rpc error: code = NotFound desc = could not find container \"1b0837907c008c4627848d0bf4d6a358ab648cfafde00eced96af0512b8d7a6f\": container with ID starting with 1b0837907c008c4627848d0bf4d6a358ab648cfafde00eced96af0512b8d7a6f not found: ID does not exist"
Sep 10 17:28:10.133307 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.133285    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a38850ba-1f03-4937-8948-07de844bc2eb-console-serving-cert\") pod \"a38850ba-1f03-4937-8948-07de844bc2eb\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") "
Sep 10 17:28:10.133433 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.133314    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-console-config\") pod \"a38850ba-1f03-4937-8948-07de844bc2eb\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") "
Sep 10 17:28:10.133433 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.133366    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-trusted-ca-bundle\") pod \"a38850ba-1f03-4937-8948-07de844bc2eb\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") "
Sep 10 17:28:10.133433 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.133400    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjlk9\" (UniqueName: \"kubernetes.io/projected/a38850ba-1f03-4937-8948-07de844bc2eb-kube-api-access-qjlk9\") pod \"a38850ba-1f03-4937-8948-07de844bc2eb\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") "
Sep 10 17:28:10.133433 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.133432    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a38850ba-1f03-4937-8948-07de844bc2eb-console-oauth-config\") pod \"a38850ba-1f03-4937-8948-07de844bc2eb\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") "
Sep 10 17:28:10.133667 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.133597    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-oauth-serving-cert\") pod \"a38850ba-1f03-4937-8948-07de844bc2eb\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") "
Sep 10 17:28:10.133729 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.133686    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-service-ca\") pod \"a38850ba-1f03-4937-8948-07de844bc2eb\" (UID: \"a38850ba-1f03-4937-8948-07de844bc2eb\") "
Sep 10 17:28:10.133729 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.133706    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-console-config" (OuterVolumeSpecName: "console-config") pod "a38850ba-1f03-4937-8948-07de844bc2eb" (UID: "a38850ba-1f03-4937-8948-07de844bc2eb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:28:10.133839 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.133821    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a38850ba-1f03-4937-8948-07de844bc2eb" (UID: "a38850ba-1f03-4937-8948-07de844bc2eb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:28:10.133989 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.133963    2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-console-config\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:28:10.134054 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.133992    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-trusted-ca-bundle\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:28:10.134054 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.133997    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-service-ca" (OuterVolumeSpecName: "service-ca") pod "a38850ba-1f03-4937-8948-07de844bc2eb" (UID: "a38850ba-1f03-4937-8948-07de844bc2eb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:28:10.134186 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.134166    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a38850ba-1f03-4937-8948-07de844bc2eb" (UID: "a38850ba-1f03-4937-8948-07de844bc2eb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:28:10.135616 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.135578    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a38850ba-1f03-4937-8948-07de844bc2eb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a38850ba-1f03-4937-8948-07de844bc2eb" (UID: "a38850ba-1f03-4937-8948-07de844bc2eb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:28:10.135711 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.135625    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a38850ba-1f03-4937-8948-07de844bc2eb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a38850ba-1f03-4937-8948-07de844bc2eb" (UID: "a38850ba-1f03-4937-8948-07de844bc2eb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:28:10.135711 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.135687    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38850ba-1f03-4937-8948-07de844bc2eb-kube-api-access-qjlk9" (OuterVolumeSpecName: "kube-api-access-qjlk9") pod "a38850ba-1f03-4937-8948-07de844bc2eb" (UID: "a38850ba-1f03-4937-8948-07de844bc2eb"). InnerVolumeSpecName "kube-api-access-qjlk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:28:10.235061 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.235026    2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a38850ba-1f03-4937-8948-07de844bc2eb-console-serving-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:28:10.235061 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.235055    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjlk9\" (UniqueName: \"kubernetes.io/projected/a38850ba-1f03-4937-8948-07de844bc2eb-kube-api-access-qjlk9\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:28:10.235061 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.235064    2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a38850ba-1f03-4937-8948-07de844bc2eb-console-oauth-config\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:28:10.235287 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.235073    2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-oauth-serving-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:28:10.235287 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.235082    2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a38850ba-1f03-4937-8948-07de844bc2eb-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:28:10.443369 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.443317    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8c8ffbcdb-mmqhc"]
Sep 10 17:28:10.447169 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.447143    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8c8ffbcdb-mmqhc"]
Sep 10 17:28:10.802556 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:10.802445    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a38850ba-1f03-4937-8948-07de844bc2eb" path="/var/lib/kubelet/pods/a38850ba-1f03-4937-8948-07de844bc2eb/volumes"
Sep 10 17:28:37.346686 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:37.346643    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:28:37.363260 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:37.363227    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:28:38.252350 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:38.252319    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:28:39.897803 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:39.897768    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wnbjr"]
Sep 10 17:28:39.898311 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:39.898111    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38850ba-1f03-4937-8948-07de844bc2eb" containerName="console"
Sep 10 17:28:39.898311 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:39.898122    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38850ba-1f03-4937-8948-07de844bc2eb" containerName="console"
Sep 10 17:28:39.898311 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:39.898181    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a38850ba-1f03-4937-8948-07de844bc2eb" containerName="console"
Sep 10 17:28:39.900751 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:39.900730    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wnbjr"
Sep 10 17:28:39.903255 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:39.903232    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 17:28:39.903356 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:39.903281    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 17:28:39.920326 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:39.920297    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wnbjr"]
Sep 10 17:28:39.983805 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:39.983762    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78x5\" (UniqueName: \"kubernetes.io/projected/bfa99ce9-2502-451f-b9ed-7e1889688517-kube-api-access-m78x5\") pod \"cert-manager-webhook-6486ffd997-wnbjr\" (UID: \"bfa99ce9-2502-451f-b9ed-7e1889688517\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnbjr"
Sep 10 17:28:39.983988 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:39.983822    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfa99ce9-2502-451f-b9ed-7e1889688517-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wnbjr\" (UID: \"bfa99ce9-2502-451f-b9ed-7e1889688517\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnbjr"
Sep 10 17:28:40.085036 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:40.085000    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfa99ce9-2502-451f-b9ed-7e1889688517-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wnbjr\" (UID: \"bfa99ce9-2502-451f-b9ed-7e1889688517\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnbjr"
Sep 10 17:28:40.085241 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:40.085108    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78x5\" (UniqueName: \"kubernetes.io/projected/bfa99ce9-2502-451f-b9ed-7e1889688517-kube-api-access-m78x5\") pod \"cert-manager-webhook-6486ffd997-wnbjr\" (UID: \"bfa99ce9-2502-451f-b9ed-7e1889688517\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnbjr"
Sep 10 17:28:40.102026 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:40.101991    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78x5\" (UniqueName: \"kubernetes.io/projected/bfa99ce9-2502-451f-b9ed-7e1889688517-kube-api-access-m78x5\") pod \"cert-manager-webhook-6486ffd997-wnbjr\" (UID: \"bfa99ce9-2502-451f-b9ed-7e1889688517\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnbjr"
Sep 10 17:28:40.102026 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:40.102008    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfa99ce9-2502-451f-b9ed-7e1889688517-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wnbjr\" (UID: \"bfa99ce9-2502-451f-b9ed-7e1889688517\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnbjr"
Sep 10 17:28:40.223810 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:40.223730    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wnbjr"
Sep 10 17:28:40.495644 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:40.495619    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wnbjr"]
Sep 10 17:28:40.497673 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:28:40.497645    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfa99ce9_2502_451f_b9ed_7e1889688517.slice/crio-de83d10ee25e2bfed12bc3c673c62d2d7c118934dcd73b476e76190ecc1cca88 WatchSource:0}: Error finding container de83d10ee25e2bfed12bc3c673c62d2d7c118934dcd73b476e76190ecc1cca88: Status 404 returned error can't find the container with id de83d10ee25e2bfed12bc3c673c62d2d7c118934dcd73b476e76190ecc1cca88
Sep 10 17:28:41.250931 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:41.250896    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wnbjr" event={"ID":"bfa99ce9-2502-451f-b9ed-7e1889688517","Type":"ContainerStarted","Data":"de83d10ee25e2bfed12bc3c673c62d2d7c118934dcd73b476e76190ecc1cca88"}
Sep 10 17:28:44.267487 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:44.267452    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wnbjr" event={"ID":"bfa99ce9-2502-451f-b9ed-7e1889688517","Type":"ContainerStarted","Data":"39d561230501b04a466eb24d0796a3248965ed8cfd2ed575a1fc75747bceedd1"}
Sep 10 17:28:44.267947 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:44.267691    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wnbjr"
Sep 10 17:28:44.287813 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:44.287760    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-wnbjr" podStartSLOduration=2.103258181 podStartE2EDuration="5.287743803s" totalImagesPullingTime="3.184485622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:28:44.286641209 +0000 UTC m=+296.045168280" watchObservedRunningTime="2026-09-10 17:28:44.287743803 +0000 UTC m=+296.046270873"
Sep 10 17:28:48.623059 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:48.623030    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:28:48.623859 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:48.623833    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:28:48.634678 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:48.634354    2552 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 17:28:50.273968 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:28:50.273933    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wnbjr"
Sep 10 17:29:18.134673 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.134637    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"]
Sep 10 17:29:18.139207 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.139186    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.143718 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.143692    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 17:29:18.143844 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.143830    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hzpgd\""
Sep 10 17:29:18.144154 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.144136    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 17:29:18.144204 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.144150    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 17:29:18.144255 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.144240    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 17:29:18.158346 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.158310    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"]
Sep 10 17:29:18.318350 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.318307    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7355930c-a88e-4edf-8eec-a8811ae65036-manifests\") pod \"opendatahub-operator-controller-manager-b6bb7766-wmbpk\" (UID: \"7355930c-a88e-4edf-8eec-a8811ae65036\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.318350 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.318356    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twz59\" (UniqueName: \"kubernetes.io/projected/7355930c-a88e-4edf-8eec-a8811ae65036-kube-api-access-twz59\") pod \"opendatahub-operator-controller-manager-b6bb7766-wmbpk\" (UID: \"7355930c-a88e-4edf-8eec-a8811ae65036\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.318579 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.318384    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7355930c-a88e-4edf-8eec-a8811ae65036-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b6bb7766-wmbpk\" (UID: \"7355930c-a88e-4edf-8eec-a8811ae65036\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.318579 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.318454    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7355930c-a88e-4edf-8eec-a8811ae65036-webhook-cert\") pod \"opendatahub-operator-controller-manager-b6bb7766-wmbpk\" (UID: \"7355930c-a88e-4edf-8eec-a8811ae65036\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.318579 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.318487    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7355930c-a88e-4edf-8eec-a8811ae65036-tmp\") pod \"opendatahub-operator-controller-manager-b6bb7766-wmbpk\" (UID: \"7355930c-a88e-4edf-8eec-a8811ae65036\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.419290 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.419206    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7355930c-a88e-4edf-8eec-a8811ae65036-webhook-cert\") pod \"opendatahub-operator-controller-manager-b6bb7766-wmbpk\" (UID: \"7355930c-a88e-4edf-8eec-a8811ae65036\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.419290 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.419238    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7355930c-a88e-4edf-8eec-a8811ae65036-tmp\") pod \"opendatahub-operator-controller-manager-b6bb7766-wmbpk\" (UID: \"7355930c-a88e-4edf-8eec-a8811ae65036\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.419552 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.419293    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7355930c-a88e-4edf-8eec-a8811ae65036-manifests\") pod \"opendatahub-operator-controller-manager-b6bb7766-wmbpk\" (UID: \"7355930c-a88e-4edf-8eec-a8811ae65036\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.419552 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.419317    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twz59\" (UniqueName: \"kubernetes.io/projected/7355930c-a88e-4edf-8eec-a8811ae65036-kube-api-access-twz59\") pod \"opendatahub-operator-controller-manager-b6bb7766-wmbpk\" (UID: \"7355930c-a88e-4edf-8eec-a8811ae65036\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.419552 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.419339    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7355930c-a88e-4edf-8eec-a8811ae65036-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b6bb7766-wmbpk\" (UID: \"7355930c-a88e-4edf-8eec-a8811ae65036\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.419836 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.419814    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7355930c-a88e-4edf-8eec-a8811ae65036-manifests\") pod \"opendatahub-operator-controller-manager-b6bb7766-wmbpk\" (UID: \"7355930c-a88e-4edf-8eec-a8811ae65036\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.421666 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.421640    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7355930c-a88e-4edf-8eec-a8811ae65036-tmp\") pod \"opendatahub-operator-controller-manager-b6bb7766-wmbpk\" (UID: \"7355930c-a88e-4edf-8eec-a8811ae65036\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.421763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.421692    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7355930c-a88e-4edf-8eec-a8811ae65036-webhook-cert\") pod \"opendatahub-operator-controller-manager-b6bb7766-wmbpk\" (UID: \"7355930c-a88e-4edf-8eec-a8811ae65036\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.421763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.421745    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7355930c-a88e-4edf-8eec-a8811ae65036-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b6bb7766-wmbpk\" (UID: \"7355930c-a88e-4edf-8eec-a8811ae65036\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.430825 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.430796    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twz59\" (UniqueName: \"kubernetes.io/projected/7355930c-a88e-4edf-8eec-a8811ae65036-kube-api-access-twz59\") pod \"opendatahub-operator-controller-manager-b6bb7766-wmbpk\" (UID: \"7355930c-a88e-4edf-8eec-a8811ae65036\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.449694 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.449664    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:18.725477 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.725323    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"]
Sep 10 17:29:18.728643 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:29:18.728613    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7355930c_a88e_4edf_8eec_a8811ae65036.slice/crio-257fa7503415898216bbd120131454b41ddcedb161905036d44c26ea585c7236 WatchSource:0}: Error finding container 257fa7503415898216bbd120131454b41ddcedb161905036d44c26ea585c7236: Status 404 returned error can't find the container with id 257fa7503415898216bbd120131454b41ddcedb161905036d44c26ea585c7236
Sep 10 17:29:18.730620 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:18.730601    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:29:19.415925 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:19.415879    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk" event={"ID":"7355930c-a88e-4edf-8eec-a8811ae65036","Type":"ContainerStarted","Data":"257fa7503415898216bbd120131454b41ddcedb161905036d44c26ea585c7236"}
Sep 10 17:29:22.433509 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:22.433470    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk" event={"ID":"7355930c-a88e-4edf-8eec-a8811ae65036","Type":"ContainerStarted","Data":"64dfadbcb244aa30de3e6e7da8e9d780be8292f18f553c6f6924c0bc3420e419"}
Sep 10 17:29:23.439468 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:23.439434    2552 generic.go:356] "Generic (PLEG): container finished" podID="7355930c-a88e-4edf-8eec-a8811ae65036" containerID="64dfadbcb244aa30de3e6e7da8e9d780be8292f18f553c6f6924c0bc3420e419" exitCode=0
Sep 10 17:29:23.439951 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:23.439512    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk" event={"ID":"7355930c-a88e-4edf-8eec-a8811ae65036","Type":"ContainerDied","Data":"64dfadbcb244aa30de3e6e7da8e9d780be8292f18f553c6f6924c0bc3420e419"}
Sep 10 17:29:24.445731 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:24.445696    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk" event={"ID":"7355930c-a88e-4edf-8eec-a8811ae65036","Type":"ContainerStarted","Data":"d89cbf72088e9677e9ae8d50e8e000afc20b0edc5a37d35da014252d52176033"}
Sep 10 17:29:24.446156 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:24.445934    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:24.467587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:24.467530    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk" podStartSLOduration=2.087111573 podStartE2EDuration="6.467486937s" totalImagesPullingTime="3.375149355s" totalInitContainerRuntime="1.005226009s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:24.465751336 +0000 UTC m=+336.224278405" watchObservedRunningTime="2026-09-10 17:29:24.467486937 +0000 UTC m=+336.226014009"
Sep 10 17:29:35.452677 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:35.452646    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-wmbpk"
Sep 10 17:29:37.157072 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.157028    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"]
Sep 10 17:29:37.165261 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.165237    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:29:37.168311 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.168286    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:29:37.168599 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.168582    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8hs6s\""
Sep 10 17:29:37.168656 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.168607    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 17:29:37.169453 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.169431    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 17:29:37.169565 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.169433    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 17:29:37.169565 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.169465    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 17:29:37.178226 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.178199    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"]
Sep 10 17:29:37.291214 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.291181    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca0845e0-a958-4783-aa8d-3a1455aa0337-metrics-cert\") pod \"lws-controller-manager-86c5cb66bd-w4qw9\" (UID: \"ca0845e0-a958-4783-aa8d-3a1455aa0337\") " pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:29:37.291390 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.291223    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ca0845e0-a958-4783-aa8d-3a1455aa0337-manager-config\") pod \"lws-controller-manager-86c5cb66bd-w4qw9\" (UID: \"ca0845e0-a958-4783-aa8d-3a1455aa0337\") " pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:29:37.291390 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.291281    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86dd2\" (UniqueName: \"kubernetes.io/projected/ca0845e0-a958-4783-aa8d-3a1455aa0337-kube-api-access-86dd2\") pod \"lws-controller-manager-86c5cb66bd-w4qw9\" (UID: \"ca0845e0-a958-4783-aa8d-3a1455aa0337\") " pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:29:37.291390 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.291333    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca0845e0-a958-4783-aa8d-3a1455aa0337-cert\") pod \"lws-controller-manager-86c5cb66bd-w4qw9\" (UID: \"ca0845e0-a958-4783-aa8d-3a1455aa0337\") " pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:29:37.392793 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.392753    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca0845e0-a958-4783-aa8d-3a1455aa0337-cert\") pod \"lws-controller-manager-86c5cb66bd-w4qw9\" (UID: \"ca0845e0-a958-4783-aa8d-3a1455aa0337\") " pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:29:37.392993 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.392801    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca0845e0-a958-4783-aa8d-3a1455aa0337-metrics-cert\") pod \"lws-controller-manager-86c5cb66bd-w4qw9\" (UID: \"ca0845e0-a958-4783-aa8d-3a1455aa0337\") " pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:29:37.392993 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.392830    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ca0845e0-a958-4783-aa8d-3a1455aa0337-manager-config\") pod \"lws-controller-manager-86c5cb66bd-w4qw9\" (UID: \"ca0845e0-a958-4783-aa8d-3a1455aa0337\") " pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:29:37.393119 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.393004    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86dd2\" (UniqueName: \"kubernetes.io/projected/ca0845e0-a958-4783-aa8d-3a1455aa0337-kube-api-access-86dd2\") pod \"lws-controller-manager-86c5cb66bd-w4qw9\" (UID: \"ca0845e0-a958-4783-aa8d-3a1455aa0337\") " pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:29:37.393613 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.393590    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ca0845e0-a958-4783-aa8d-3a1455aa0337-manager-config\") pod \"lws-controller-manager-86c5cb66bd-w4qw9\" (UID: \"ca0845e0-a958-4783-aa8d-3a1455aa0337\") " pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:29:37.395428 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.395402    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca0845e0-a958-4783-aa8d-3a1455aa0337-metrics-cert\") pod \"lws-controller-manager-86c5cb66bd-w4qw9\" (UID: \"ca0845e0-a958-4783-aa8d-3a1455aa0337\") " pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:29:37.395549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.395441    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca0845e0-a958-4783-aa8d-3a1455aa0337-cert\") pod \"lws-controller-manager-86c5cb66bd-w4qw9\" (UID: \"ca0845e0-a958-4783-aa8d-3a1455aa0337\") " pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:29:37.420662 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.420598    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86dd2\" (UniqueName: \"kubernetes.io/projected/ca0845e0-a958-4783-aa8d-3a1455aa0337-kube-api-access-86dd2\") pod \"lws-controller-manager-86c5cb66bd-w4qw9\" (UID: \"ca0845e0-a958-4783-aa8d-3a1455aa0337\") " pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:29:37.474218 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.474187    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:29:37.761305 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:37.761275    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"]
Sep 10 17:29:37.763806 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:29:37.763780    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0845e0_a958_4783_aa8d_3a1455aa0337.slice/crio-764e65ddd2c78a315502b8c29f102cd25354df36445229c2762db423b3874d5b WatchSource:0}: Error finding container 764e65ddd2c78a315502b8c29f102cd25354df36445229c2762db423b3874d5b: Status 404 returned error can't find the container with id 764e65ddd2c78a315502b8c29f102cd25354df36445229c2762db423b3874d5b
Sep 10 17:29:38.123985 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.123943    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr"]
Sep 10 17:29:38.131305 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.131281    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr"
Sep 10 17:29:38.134195 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.134168    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 17:29:38.134326 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.134255    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-tqw57\""
Sep 10 17:29:38.134326 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.134274    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 17:29:38.134446 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.134344    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 17:29:38.135287 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.135268    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 17:29:38.146602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.146575    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr"]
Sep 10 17:29:38.201447 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.201415    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsm72\" (UniqueName: \"kubernetes.io/projected/6fd72f14-7331-4679-9cf6-c29e3c9222f6-kube-api-access-xsm72\") pod \"kube-auth-proxy-7bf54bbc88-gb5nr\" (UID: \"6fd72f14-7331-4679-9cf6-c29e3c9222f6\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr"
Sep 10 17:29:38.201909 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.201467    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd72f14-7331-4679-9cf6-c29e3c9222f6-tls-certs\") pod \"kube-auth-proxy-7bf54bbc88-gb5nr\" (UID: \"6fd72f14-7331-4679-9cf6-c29e3c9222f6\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr"
Sep 10 17:29:38.201909 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.201581    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fd72f14-7331-4679-9cf6-c29e3c9222f6-tmp\") pod \"kube-auth-proxy-7bf54bbc88-gb5nr\" (UID: \"6fd72f14-7331-4679-9cf6-c29e3c9222f6\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr"
Sep 10 17:29:38.302882 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.302847    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fd72f14-7331-4679-9cf6-c29e3c9222f6-tmp\") pod \"kube-auth-proxy-7bf54bbc88-gb5nr\" (UID: \"6fd72f14-7331-4679-9cf6-c29e3c9222f6\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr"
Sep 10 17:29:38.303086 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.302910    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsm72\" (UniqueName: \"kubernetes.io/projected/6fd72f14-7331-4679-9cf6-c29e3c9222f6-kube-api-access-xsm72\") pod \"kube-auth-proxy-7bf54bbc88-gb5nr\" (UID: \"6fd72f14-7331-4679-9cf6-c29e3c9222f6\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr"
Sep 10 17:29:38.303086 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.302955    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd72f14-7331-4679-9cf6-c29e3c9222f6-tls-certs\") pod \"kube-auth-proxy-7bf54bbc88-gb5nr\" (UID: \"6fd72f14-7331-4679-9cf6-c29e3c9222f6\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr"
Sep 10 17:29:38.303204 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:29:38.303091    2552 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 17:29:38.303204 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:29:38.303156    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fd72f14-7331-4679-9cf6-c29e3c9222f6-tls-certs podName:6fd72f14-7331-4679-9cf6-c29e3c9222f6 nodeName:}" failed. No retries permitted until 2026-09-10 17:29:38.803137621 +0000 UTC m=+350.561664683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6fd72f14-7331-4679-9cf6-c29e3c9222f6-tls-certs") pod "kube-auth-proxy-7bf54bbc88-gb5nr" (UID: "6fd72f14-7331-4679-9cf6-c29e3c9222f6") : secret "kube-auth-proxy-tls" not found
Sep 10 17:29:38.305327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.305299    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fd72f14-7331-4679-9cf6-c29e3c9222f6-tmp\") pod \"kube-auth-proxy-7bf54bbc88-gb5nr\" (UID: \"6fd72f14-7331-4679-9cf6-c29e3c9222f6\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr"
Sep 10 17:29:38.318656 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.318633    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsm72\" (UniqueName: \"kubernetes.io/projected/6fd72f14-7331-4679-9cf6-c29e3c9222f6-kube-api-access-xsm72\") pod \"kube-auth-proxy-7bf54bbc88-gb5nr\" (UID: \"6fd72f14-7331-4679-9cf6-c29e3c9222f6\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr"
Sep 10 17:29:38.508005 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.507922    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9" event={"ID":"ca0845e0-a958-4783-aa8d-3a1455aa0337","Type":"ContainerStarted","Data":"764e65ddd2c78a315502b8c29f102cd25354df36445229c2762db423b3874d5b"}
Sep 10 17:29:38.807898 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.807813    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd72f14-7331-4679-9cf6-c29e3c9222f6-tls-certs\") pod \"kube-auth-proxy-7bf54bbc88-gb5nr\" (UID: \"6fd72f14-7331-4679-9cf6-c29e3c9222f6\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr"
Sep 10 17:29:38.811237 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:38.810971    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd72f14-7331-4679-9cf6-c29e3c9222f6-tls-certs\") pod \"kube-auth-proxy-7bf54bbc88-gb5nr\" (UID: \"6fd72f14-7331-4679-9cf6-c29e3c9222f6\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr"
Sep 10 17:29:39.041615 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:39.041575    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr"
Sep 10 17:29:39.346048 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:39.345934    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr"]
Sep 10 17:29:40.234174 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:29:40.234135    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd72f14_7331_4679_9cf6_c29e3c9222f6.slice/crio-b5ab5593795f2250db0ecf489794075f79f41b29b83d33da7e9aa7bc9927212f WatchSource:0}: Error finding container b5ab5593795f2250db0ecf489794075f79f41b29b83d33da7e9aa7bc9927212f: Status 404 returned error can't find the container with id b5ab5593795f2250db0ecf489794075f79f41b29b83d33da7e9aa7bc9927212f
Sep 10 17:29:40.519141 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:40.519051    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr" event={"ID":"6fd72f14-7331-4679-9cf6-c29e3c9222f6","Type":"ContainerStarted","Data":"b5ab5593795f2250db0ecf489794075f79f41b29b83d33da7e9aa7bc9927212f"}
Sep 10 17:29:40.520657 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:40.520631    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9" event={"ID":"ca0845e0-a958-4783-aa8d-3a1455aa0337","Type":"ContainerStarted","Data":"9a05e75bfbaeb9129aee1b6e89213049fb307f1df7c09f8e94830d999ac4d2c8"}
Sep 10 17:29:40.520768 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:40.520750    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:29:40.537402 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:40.537336    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9" podStartSLOduration=1.00894554 podStartE2EDuration="3.537318827s" totalImagesPullingTime="2.528373287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:40.536354731 +0000 UTC m=+352.294881794" watchObservedRunningTime="2026-09-10 17:29:40.537318827 +0000 UTC m=+352.295845896"
Sep 10 17:29:43.537790 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:43.537752    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr" event={"ID":"6fd72f14-7331-4679-9cf6-c29e3c9222f6","Type":"ContainerStarted","Data":"adeb94cf5fe842fc7e509adad0074a1353d981f8173c3fc7b297810309670362"}
Sep 10 17:29:43.556208 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:43.556153    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-gb5nr" podStartSLOduration=3.027526034 podStartE2EDuration="5.5561364s" totalImagesPullingTime="2.528610366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:43.555157488 +0000 UTC m=+355.313684559" watchObservedRunningTime="2026-09-10 17:29:43.5561364 +0000 UTC m=+355.314663469"
Sep 10 17:29:51.529400 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:29:51.529367    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-86c5cb66bd-w4qw9"
Sep 10 17:31:40.380807 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.380772    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5"]
Sep 10 17:31:40.384328 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.384307    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5"
Sep 10 17:31:40.387069 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.387048    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8jzgf\""
Sep 10 17:31:40.387173 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.387088    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 17:31:40.388257 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.388240    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 17:31:40.388346 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.388283    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 17:31:40.388346 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.388340    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 17:31:40.394971 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.394952    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5"]
Sep 10 17:31:40.546035 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.545995    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3e67cb1b-0e95-45db-9d3b-979e933acfbf-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-btmx5\" (UID: \"3e67cb1b-0e95-45db-9d3b-979e933acfbf\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5"
Sep 10 17:31:40.546202 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.546064    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m977w\" (UniqueName: \"kubernetes.io/projected/3e67cb1b-0e95-45db-9d3b-979e933acfbf-kube-api-access-m977w\") pod \"kuadrant-console-plugin-fbdd4f445-btmx5\" (UID: \"3e67cb1b-0e95-45db-9d3b-979e933acfbf\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5"
Sep 10 17:31:40.546202 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.546116    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e67cb1b-0e95-45db-9d3b-979e933acfbf-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-btmx5\" (UID: \"3e67cb1b-0e95-45db-9d3b-979e933acfbf\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5"
Sep 10 17:31:40.647606 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.647485    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3e67cb1b-0e95-45db-9d3b-979e933acfbf-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-btmx5\" (UID: \"3e67cb1b-0e95-45db-9d3b-979e933acfbf\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5"
Sep 10 17:31:40.647606 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.647577    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m977w\" (UniqueName: \"kubernetes.io/projected/3e67cb1b-0e95-45db-9d3b-979e933acfbf-kube-api-access-m977w\") pod \"kuadrant-console-plugin-fbdd4f445-btmx5\" (UID: \"3e67cb1b-0e95-45db-9d3b-979e933acfbf\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5"
Sep 10 17:31:40.647606 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.647601    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e67cb1b-0e95-45db-9d3b-979e933acfbf-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-btmx5\" (UID: \"3e67cb1b-0e95-45db-9d3b-979e933acfbf\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5"
Sep 10 17:31:40.647872 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:31:40.647739    2552 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 10 17:31:40.647872 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:31:40.647799    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e67cb1b-0e95-45db-9d3b-979e933acfbf-plugin-serving-cert podName:3e67cb1b-0e95-45db-9d3b-979e933acfbf nodeName:}" failed. No retries permitted until 2026-09-10 17:31:41.147781385 +0000 UTC m=+472.906308433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/3e67cb1b-0e95-45db-9d3b-979e933acfbf-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-btmx5" (UID: "3e67cb1b-0e95-45db-9d3b-979e933acfbf") : secret "plugin-serving-cert" not found
Sep 10 17:31:40.648231 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.648203    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3e67cb1b-0e95-45db-9d3b-979e933acfbf-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-btmx5\" (UID: \"3e67cb1b-0e95-45db-9d3b-979e933acfbf\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5"
Sep 10 17:31:40.659279 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:40.659246    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m977w\" (UniqueName: \"kubernetes.io/projected/3e67cb1b-0e95-45db-9d3b-979e933acfbf-kube-api-access-m977w\") pod \"kuadrant-console-plugin-fbdd4f445-btmx5\" (UID: \"3e67cb1b-0e95-45db-9d3b-979e933acfbf\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5"
Sep 10 17:31:41.153036 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:41.152998    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e67cb1b-0e95-45db-9d3b-979e933acfbf-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-btmx5\" (UID: \"3e67cb1b-0e95-45db-9d3b-979e933acfbf\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5"
Sep 10 17:31:41.155387 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:41.155355    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e67cb1b-0e95-45db-9d3b-979e933acfbf-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-btmx5\" (UID: \"3e67cb1b-0e95-45db-9d3b-979e933acfbf\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5"
Sep 10 17:31:41.296053 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:41.296005    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5"
Sep 10 17:31:41.568877 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:41.568850    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5"]
Sep 10 17:31:41.570984 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:31:41.570953    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e67cb1b_0e95_45db_9d3b_979e933acfbf.slice/crio-d9237bcbff8e7b2ff0ef16138704e0cce5b541526f77e55390c8dd6725a3b849 WatchSource:0}: Error finding container d9237bcbff8e7b2ff0ef16138704e0cce5b541526f77e55390c8dd6725a3b849: Status 404 returned error can't find the container with id d9237bcbff8e7b2ff0ef16138704e0cce5b541526f77e55390c8dd6725a3b849
Sep 10 17:31:42.066341 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:42.066307    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5" event={"ID":"3e67cb1b-0e95-45db-9d3b-979e933acfbf","Type":"ContainerStarted","Data":"d9237bcbff8e7b2ff0ef16138704e0cce5b541526f77e55390c8dd6725a3b849"}
Sep 10 17:31:46.096191 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:46.096151    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5" event={"ID":"3e67cb1b-0e95-45db-9d3b-979e933acfbf","Type":"ContainerStarted","Data":"124f0c86ec97d689a0df2e41b763b6b80c23778f16a98415855f7da301c9da14"}
Sep 10 17:31:46.115446 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:31:46.115391    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btmx5" podStartSLOduration=1.7033664750000002 podStartE2EDuration="6.115370678s" totalImagesPullingTime="4.412004203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:31:46.113578085 +0000 UTC m=+477.872105155" watchObservedRunningTime="2026-09-10 17:31:46.115370678 +0000 UTC m=+477.873897751"
Sep 10 17:32:23.973284 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:23.973244    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:32:23.990931 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:23.990904    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:32:23.991090 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:23.990992    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8565f"
Sep 10 17:32:23.993420 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:23.993389    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 17:32:23.999228 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:23.999201    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:32:24.115424 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.115390    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44ec4c7e-fc85-4d18-a150-bc77a6b1a65c-config-file\") pod \"limitador-limitador-84994f7ffc-8565f\" (UID: \"44ec4c7e-fc85-4d18-a150-bc77a6b1a65c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8565f"
Sep 10 17:32:24.115613 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.115434    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5299n\" (UniqueName: \"kubernetes.io/projected/44ec4c7e-fc85-4d18-a150-bc77a6b1a65c-kube-api-access-5299n\") pod \"limitador-limitador-84994f7ffc-8565f\" (UID: \"44ec4c7e-fc85-4d18-a150-bc77a6b1a65c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8565f"
Sep 10 17:32:24.216593 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.216559    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44ec4c7e-fc85-4d18-a150-bc77a6b1a65c-config-file\") pod \"limitador-limitador-84994f7ffc-8565f\" (UID: \"44ec4c7e-fc85-4d18-a150-bc77a6b1a65c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8565f"
Sep 10 17:32:24.216731 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.216606    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5299n\" (UniqueName: \"kubernetes.io/projected/44ec4c7e-fc85-4d18-a150-bc77a6b1a65c-kube-api-access-5299n\") pod \"limitador-limitador-84994f7ffc-8565f\" (UID: \"44ec4c7e-fc85-4d18-a150-bc77a6b1a65c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8565f"
Sep 10 17:32:24.217209 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.217189    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44ec4c7e-fc85-4d18-a150-bc77a6b1a65c-config-file\") pod \"limitador-limitador-84994f7ffc-8565f\" (UID: \"44ec4c7e-fc85-4d18-a150-bc77a6b1a65c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8565f"
Sep 10 17:32:24.226394 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.226339    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5299n\" (UniqueName: \"kubernetes.io/projected/44ec4c7e-fc85-4d18-a150-bc77a6b1a65c-kube-api-access-5299n\") pod \"limitador-limitador-84994f7ffc-8565f\" (UID: \"44ec4c7e-fc85-4d18-a150-bc77a6b1a65c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8565f"
Sep 10 17:32:24.301593 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.301556    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8565f"
Sep 10 17:32:24.586139 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.586115    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:32:24.587372 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:32:24.587343    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ec4c7e_fc85_4d18_a150_bc77a6b1a65c.slice/crio-72dd4c3aa19f21c76a27fed60de3b9d7f9de615448e7263ce90859441f4df6f8 WatchSource:0}: Error finding container 72dd4c3aa19f21c76a27fed60de3b9d7f9de615448e7263ce90859441f4df6f8: Status 404 returned error can't find the container with id 72dd4c3aa19f21c76a27fed60de3b9d7f9de615448e7263ce90859441f4df6f8
Sep 10 17:32:24.637685 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.637657    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-6lhb8"]
Sep 10 17:32:24.641910 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.641894    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6lhb8"
Sep 10 17:32:24.645111 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.645064    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q6dps\""
Sep 10 17:32:24.647815 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.647796    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6lhb8"]
Sep 10 17:32:24.722082 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.722052    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t5pk\" (UniqueName: \"kubernetes.io/projected/e4d4660e-7e8e-48db-ad3a-dd5d25a3981b-kube-api-access-6t5pk\") pod \"authorino-59c974bd48-6lhb8\" (UID: \"e4d4660e-7e8e-48db-ad3a-dd5d25a3981b\") " pod="kuadrant-system/authorino-59c974bd48-6lhb8"
Sep 10 17:32:24.823342 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.823308    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t5pk\" (UniqueName: \"kubernetes.io/projected/e4d4660e-7e8e-48db-ad3a-dd5d25a3981b-kube-api-access-6t5pk\") pod \"authorino-59c974bd48-6lhb8\" (UID: \"e4d4660e-7e8e-48db-ad3a-dd5d25a3981b\") " pod="kuadrant-system/authorino-59c974bd48-6lhb8"
Sep 10 17:32:24.834143 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.834079    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t5pk\" (UniqueName: \"kubernetes.io/projected/e4d4660e-7e8e-48db-ad3a-dd5d25a3981b-kube-api-access-6t5pk\") pod \"authorino-59c974bd48-6lhb8\" (UID: \"e4d4660e-7e8e-48db-ad3a-dd5d25a3981b\") " pod="kuadrant-system/authorino-59c974bd48-6lhb8"
Sep 10 17:32:24.951854 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:24.951816    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6lhb8"
Sep 10 17:32:25.236884 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:25.236763    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6lhb8"]
Sep 10 17:32:25.241420 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:32:25.241385    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4d4660e_7e8e_48db_ad3a_dd5d25a3981b.slice/crio-afad77746a72b4a919c7350ce0c41f81dca9334c95d0e7f01a2a3c596fb38c21 WatchSource:0}: Error finding container afad77746a72b4a919c7350ce0c41f81dca9334c95d0e7f01a2a3c596fb38c21: Status 404 returned error can't find the container with id afad77746a72b4a919c7350ce0c41f81dca9334c95d0e7f01a2a3c596fb38c21
Sep 10 17:32:25.275550 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:25.275478    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6lhb8" event={"ID":"e4d4660e-7e8e-48db-ad3a-dd5d25a3981b","Type":"ContainerStarted","Data":"afad77746a72b4a919c7350ce0c41f81dca9334c95d0e7f01a2a3c596fb38c21"}
Sep 10 17:32:25.276881 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:25.276854    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8565f" event={"ID":"44ec4c7e-fc85-4d18-a150-bc77a6b1a65c","Type":"ContainerStarted","Data":"72dd4c3aa19f21c76a27fed60de3b9d7f9de615448e7263ce90859441f4df6f8"}
Sep 10 17:32:28.295440 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:28.295403    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6lhb8" event={"ID":"e4d4660e-7e8e-48db-ad3a-dd5d25a3981b","Type":"ContainerStarted","Data":"520490add7c0e5f675bb869837deefd44629455f2d4dbaefc01392a64993a13e"}
Sep 10 17:32:28.296979 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:28.296957    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8565f" event={"ID":"44ec4c7e-fc85-4d18-a150-bc77a6b1a65c","Type":"ContainerStarted","Data":"a9b837cae6ba69c718adc9351098e55a511536492a7cceea786775bf3977fdfb"}
Sep 10 17:32:28.297108 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:28.297036    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8565f"
Sep 10 17:32:28.311446 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:28.311403    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-6lhb8" podStartSLOduration=2.107136806 podStartE2EDuration="4.311390863s" totalImagesPullingTime="2.204254057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:32:28.310958963 +0000 UTC m=+520.069486032" watchObservedRunningTime="2026-09-10 17:32:28.311390863 +0000 UTC m=+520.069917933"
Sep 10 17:32:28.329705 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:28.329648    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-8565f" podStartSLOduration=2.478677326 podStartE2EDuration="5.329630126s" totalImagesPullingTime="2.8509528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:32:28.327045022 +0000 UTC m=+520.085572094" watchObservedRunningTime="2026-09-10 17:32:28.329630126 +0000 UTC m=+520.088157198"
Sep 10 17:32:39.302255 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:39.302225    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8565f"
Sep 10 17:32:48.666379 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:48.666348    2552 scope.go:117] "RemoveContainer" containerID="662201824d9cc5fb7fb86ccc7a582f36d27acce201071b95e9a38d15dd8542f8"
Sep 10 17:32:59.137338 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:59.137302    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6lhb8"]
Sep 10 17:32:59.137934 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:59.137559    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-6lhb8" podUID="e4d4660e-7e8e-48db-ad3a-dd5d25a3981b" containerName="authorino" containerID="cri-o://520490add7c0e5f675bb869837deefd44629455f2d4dbaefc01392a64993a13e" gracePeriod=30
Sep 10 17:32:59.383237 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:59.383212    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6lhb8"
Sep 10 17:32:59.428693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:59.428613    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t5pk\" (UniqueName: \"kubernetes.io/projected/e4d4660e-7e8e-48db-ad3a-dd5d25a3981b-kube-api-access-6t5pk\") pod \"e4d4660e-7e8e-48db-ad3a-dd5d25a3981b\" (UID: \"e4d4660e-7e8e-48db-ad3a-dd5d25a3981b\") "
Sep 10 17:32:59.430787 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:59.430761    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d4660e-7e8e-48db-ad3a-dd5d25a3981b-kube-api-access-6t5pk" (OuterVolumeSpecName: "kube-api-access-6t5pk") pod "e4d4660e-7e8e-48db-ad3a-dd5d25a3981b" (UID: "e4d4660e-7e8e-48db-ad3a-dd5d25a3981b"). InnerVolumeSpecName "kube-api-access-6t5pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:32:59.444085 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:59.444053    2552 generic.go:356] "Generic (PLEG): container finished" podID="e4d4660e-7e8e-48db-ad3a-dd5d25a3981b" containerID="520490add7c0e5f675bb869837deefd44629455f2d4dbaefc01392a64993a13e" exitCode=0
Sep 10 17:32:59.444190 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:59.444104    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6lhb8"
Sep 10 17:32:59.444190 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:59.444114    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6lhb8" event={"ID":"e4d4660e-7e8e-48db-ad3a-dd5d25a3981b","Type":"ContainerDied","Data":"520490add7c0e5f675bb869837deefd44629455f2d4dbaefc01392a64993a13e"}
Sep 10 17:32:59.444190 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:59.444149    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6lhb8" event={"ID":"e4d4660e-7e8e-48db-ad3a-dd5d25a3981b","Type":"ContainerDied","Data":"afad77746a72b4a919c7350ce0c41f81dca9334c95d0e7f01a2a3c596fb38c21"}
Sep 10 17:32:59.444190 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:59.444170    2552 scope.go:117] "RemoveContainer" containerID="520490add7c0e5f675bb869837deefd44629455f2d4dbaefc01392a64993a13e"
Sep 10 17:32:59.452636 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:59.452617    2552 scope.go:117] "RemoveContainer" containerID="520490add7c0e5f675bb869837deefd44629455f2d4dbaefc01392a64993a13e"
Sep 10 17:32:59.452892 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:32:59.452874    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520490add7c0e5f675bb869837deefd44629455f2d4dbaefc01392a64993a13e\": container with ID starting with 520490add7c0e5f675bb869837deefd44629455f2d4dbaefc01392a64993a13e not found: ID does not exist" containerID="520490add7c0e5f675bb869837deefd44629455f2d4dbaefc01392a64993a13e"
Sep 10 17:32:59.453013 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:59.452898    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520490add7c0e5f675bb869837deefd44629455f2d4dbaefc01392a64993a13e"} err="failed to get container status \"520490add7c0e5f675bb869837deefd44629455f2d4dbaefc01392a64993a13e\": rpc error: code = NotFound desc = could not find container \"520490add7c0e5f675bb869837deefd44629455f2d4dbaefc01392a64993a13e\": container with ID starting with 520490add7c0e5f675bb869837deefd44629455f2d4dbaefc01392a64993a13e not found: ID does not exist"
Sep 10 17:32:59.463598 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:59.463577    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6lhb8"]
Sep 10 17:32:59.467257 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:59.467233    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6lhb8"]
Sep 10 17:32:59.529185 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:32:59.529155    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t5pk\" (UniqueName: \"kubernetes.io/projected/e4d4660e-7e8e-48db-ad3a-dd5d25a3981b-kube-api-access-6t5pk\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:33:00.802683 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:00.802651    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4d4660e-7e8e-48db-ad3a-dd5d25a3981b" path="/var/lib/kubelet/pods/e4d4660e-7e8e-48db-ad3a-dd5d25a3981b/volumes"
Sep 10 17:33:38.651931 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:38.651893    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:33:48.654415 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:48.654381    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:33:48.654862 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:48.654471    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:33:55.098248 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.098194    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"]
Sep 10 17:33:55.099782 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.098758    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4d4660e-7e8e-48db-ad3a-dd5d25a3981b" containerName="authorino"
Sep 10 17:33:55.099782 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.098774    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4d4660e-7e8e-48db-ad3a-dd5d25a3981b" containerName="authorino"
Sep 10 17:33:55.099782 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.098861    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4d4660e-7e8e-48db-ad3a-dd5d25a3981b" containerName="authorino"
Sep 10 17:33:55.100902 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.100879    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.103828 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.103801    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 17:33:55.104931 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.104891    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 10 17:33:55.105052 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.105008    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hjjt8\""
Sep 10 17:33:55.105144 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.104934    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 17:33:55.111009 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.110794    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"]
Sep 10 17:33:55.224829 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.224790    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.224829 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.224836    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.225054 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.224928    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.225054 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.224977    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhlsr\" (UniqueName: \"kubernetes.io/projected/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-kube-api-access-lhlsr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.225054 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.225030    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.225054 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.225047    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.325799 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.325758    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhlsr\" (UniqueName: \"kubernetes.io/projected/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-kube-api-access-lhlsr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.325799 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.325803    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.326020 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.325828    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.326020 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.325878    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.326020 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.325914    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.326020 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.325988    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.326286 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.326247    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.326451 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.326258    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.326451 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.326314    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.328096 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.328070    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.328479 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.328457    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.334126 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.334109    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhlsr\" (UniqueName: \"kubernetes.io/projected/9c8821e3-f435-4e87-844b-2ee6a2fea0ec-kube-api-access-lhlsr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm\" (UID: \"9c8821e3-f435-4e87-844b-2ee6a2fea0ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.413890 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.413806    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:33:55.515132 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.515096    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:33:55.683354 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.683328    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"]
Sep 10 17:33:55.686108 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:33:55.686077    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8821e3_f435_4e87_844b_2ee6a2fea0ec.slice/crio-276d71deb9c9c1b9e838b0e7f69b4d872e9c91bccfa0f59b88222e6b02a09589 WatchSource:0}: Error finding container 276d71deb9c9c1b9e838b0e7f69b4d872e9c91bccfa0f59b88222e6b02a09589: Status 404 returned error can't find the container with id 276d71deb9c9c1b9e838b0e7f69b4d872e9c91bccfa0f59b88222e6b02a09589
Sep 10 17:33:55.701593 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:55.701566    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm" event={"ID":"9c8821e3-f435-4e87-844b-2ee6a2fea0ec","Type":"ContainerStarted","Data":"276d71deb9c9c1b9e838b0e7f69b4d872e9c91bccfa0f59b88222e6b02a09589"}
Sep 10 17:33:57.711144 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:57.711110    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm" event={"ID":"9c8821e3-f435-4e87-844b-2ee6a2fea0ec","Type":"ContainerStarted","Data":"d618528dd2611adf79f7d5c0e4634ab2c613ff24b87f3f04e99fae41de55b153"}
Sep 10 17:33:57.731096 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:33:57.731049    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm" podStartSLOduration=1.705106257 podStartE2EDuration="2.731032878s" totalImagesPullingTime="1.025926621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:33:57.729052141 +0000 UTC m=+609.487579211" watchObservedRunningTime="2026-09-10 17:33:57.731032878 +0000 UTC m=+609.489559938"
Sep 10 17:34:01.193632 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.193593    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"]
Sep 10 17:34:01.196135 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.196118    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.199020 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.198998    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 17:34:01.209807 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.209786    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"]
Sep 10 17:34:01.279109 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.279072    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.279292 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.279113    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.279292 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.279151    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.279292 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.279172    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.279292 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.279208    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.279479 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.279343    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj6k4\" (UniqueName: \"kubernetes.io/projected/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-kube-api-access-dj6k4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.379840 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.379791    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj6k4\" (UniqueName: \"kubernetes.io/projected/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-kube-api-access-dj6k4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.379840 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.379845    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.380090 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.379875    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.380090 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.379912    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.380090 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.379936    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.380090 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.379972    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.380393 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.380367    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.380470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.380398    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.380470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.380415    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.382181 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.382162    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.382588 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.382569    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.391019 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.390994    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj6k4\" (UniqueName: \"kubernetes.io/projected/9e1eec56-17e5-4cb1-8628-ce81e0ea0b16-kube-api-access-dj6k4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j7l6n\" (UID: \"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.505909 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.505807    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:01.775244 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.775178    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"]
Sep 10 17:34:01.777656 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:34:01.777630    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1eec56_17e5_4cb1_8628_ce81e0ea0b16.slice/crio-6c0eb763fce7da919d098564b44e23491e1a547efa97a8ec0fa95f79b2f40aff WatchSource:0}: Error finding container 6c0eb763fce7da919d098564b44e23491e1a547efa97a8ec0fa95f79b2f40aff: Status 404 returned error can't find the container with id 6c0eb763fce7da919d098564b44e23491e1a547efa97a8ec0fa95f79b2f40aff
Sep 10 17:34:01.997942 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:01.997912    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:34:02.735521 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:02.735475    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n" event={"ID":"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16","Type":"ContainerStarted","Data":"76783eab6760bfa696ca29d19cb08968db905676986b4ae3db4b6840d7487fb7"}
Sep 10 17:34:02.736022 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:02.735536    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n" event={"ID":"9e1eec56-17e5-4cb1-8628-ce81e0ea0b16","Type":"ContainerStarted","Data":"6c0eb763fce7da919d098564b44e23491e1a547efa97a8ec0fa95f79b2f40aff"}
Sep 10 17:34:02.758848 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:02.758789    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n" podStartSLOduration=1.596612954 podStartE2EDuration="1.758771482s" totalImagesPullingTime="162.158528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:02.756888412 +0000 UTC m=+614.515415505" watchObservedRunningTime="2026-09-10 17:34:02.758771482 +0000 UTC m=+614.517298552"
Sep 10 17:34:05.414768 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:05.414725    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:34:05.415167 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:05.414791    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:34:05.427336 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:05.427302    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:34:05.760428 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:05.760341    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm"
Sep 10 17:34:11.506627 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:11.506527    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:11.506627 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:11.506590    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:11.518944 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:11.518915    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:11.788720 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:11.788631    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j7l6n"
Sep 10 17:34:23.100114 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:23.100076    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:34:32.606348 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:32.606307    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:34:39.605785 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:39.605744    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:34:49.501682 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:49.501636    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:34:57.897280 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:57.897242    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"]
Sep 10 17:34:57.900863 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:57.900840    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:57.904175 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:57.904153    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 17:34:57.919955 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:57.919929    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"]
Sep 10 17:34:58.005610 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.005579    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.005794 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.005620    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4sjb\" (UniqueName: \"kubernetes.io/projected/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-kube-api-access-v4sjb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.005794 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.005697    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.005794 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.005765    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.005915 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.005826    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.005915 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.005842    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.107280 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.107243    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4sjb\" (UniqueName: \"kubernetes.io/projected/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-kube-api-access-v4sjb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.107453 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.107298    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.107453 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.107406    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.107585 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.107514    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.107585 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.107547    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.107702 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.107608    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.107762 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.107724    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.107913 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.107888    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.107978 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.107923    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.109711 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.109692    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.109893 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.109877    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.115754 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.115731    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4sjb\" (UniqueName: \"kubernetes.io/projected/dc851fb4-2a5c-40d9-8f5f-084aee0dae96-kube-api-access-v4sjb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5\" (UID: \"dc851fb4-2a5c-40d9-8f5f-084aee0dae96\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.210537 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.210437    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:34:58.483090 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.483064    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"]
Sep 10 17:34:58.485255 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:34:58.485229    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc851fb4_2a5c_40d9_8f5f_084aee0dae96.slice/crio-6532f810cf0383a21852f17948dbd64b362e3c17aa338f8ce42cc0396d173e1d WatchSource:0}: Error finding container 6532f810cf0383a21852f17948dbd64b362e3c17aa338f8ce42cc0396d173e1d: Status 404 returned error can't find the container with id 6532f810cf0383a21852f17948dbd64b362e3c17aa338f8ce42cc0396d173e1d
Sep 10 17:34:58.486746 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:58.486725    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:34:59.004811 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:59.004728    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5" event={"ID":"dc851fb4-2a5c-40d9-8f5f-084aee0dae96","Type":"ContainerStarted","Data":"547833e6909cfca098c1f1d820b0fe2dc2b656b1d4634a89d5af422b48f1da98"}
Sep 10 17:34:59.004811 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:59.004773    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5" event={"ID":"dc851fb4-2a5c-40d9-8f5f-084aee0dae96","Type":"ContainerStarted","Data":"6532f810cf0383a21852f17948dbd64b362e3c17aa338f8ce42cc0396d173e1d"}
Sep 10 17:34:59.023402 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:34:59.023350    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5" podStartSLOduration=1.835440643 podStartE2EDuration="2.023333373s" totalImagesPullingTime="187.89273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:59.021521616 +0000 UTC m=+670.780048680" watchObservedRunningTime="2026-09-10 17:34:59.023333373 +0000 UTC m=+670.781860442"
Sep 10 17:35:00.400696 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:00.400661    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:35:08.211419 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:08.211373    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:35:08.211856 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:08.211437    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:35:08.223848 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:08.223815    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:35:09.063099 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:09.063067    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5"
Sep 10 17:35:52.021899 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:52.021862    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6bf9c7b78-bb5jp"]
Sep 10 17:35:52.025568 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:52.025549    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bf9c7b78-bb5jp"
Sep 10 17:35:52.030438 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:52.030417    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 17:35:52.031837 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:52.031625    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q6dps\""
Sep 10 17:35:52.043602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:52.043579    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bf9c7b78-bb5jp"]
Sep 10 17:35:52.086518 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:52.086465    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39eb76b8-26c0-41c7-ad67-6918247344e9-tls-cert\") pod \"authorino-6bf9c7b78-bb5jp\" (UID: \"39eb76b8-26c0-41c7-ad67-6918247344e9\") " pod="kuadrant-system/authorino-6bf9c7b78-bb5jp"
Sep 10 17:35:52.086678 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:52.086527    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5plx\" (UniqueName: \"kubernetes.io/projected/39eb76b8-26c0-41c7-ad67-6918247344e9-kube-api-access-t5plx\") pod \"authorino-6bf9c7b78-bb5jp\" (UID: \"39eb76b8-26c0-41c7-ad67-6918247344e9\") " pod="kuadrant-system/authorino-6bf9c7b78-bb5jp"
Sep 10 17:35:52.188043 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:52.188008    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39eb76b8-26c0-41c7-ad67-6918247344e9-tls-cert\") pod \"authorino-6bf9c7b78-bb5jp\" (UID: \"39eb76b8-26c0-41c7-ad67-6918247344e9\") " pod="kuadrant-system/authorino-6bf9c7b78-bb5jp"
Sep 10 17:35:52.188222 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:52.188056    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5plx\" (UniqueName: \"kubernetes.io/projected/39eb76b8-26c0-41c7-ad67-6918247344e9-kube-api-access-t5plx\") pod \"authorino-6bf9c7b78-bb5jp\" (UID: \"39eb76b8-26c0-41c7-ad67-6918247344e9\") " pod="kuadrant-system/authorino-6bf9c7b78-bb5jp"
Sep 10 17:35:52.190436 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:52.190417    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39eb76b8-26c0-41c7-ad67-6918247344e9-tls-cert\") pod \"authorino-6bf9c7b78-bb5jp\" (UID: \"39eb76b8-26c0-41c7-ad67-6918247344e9\") " pod="kuadrant-system/authorino-6bf9c7b78-bb5jp"
Sep 10 17:35:52.196227 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:52.196198    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5plx\" (UniqueName: \"kubernetes.io/projected/39eb76b8-26c0-41c7-ad67-6918247344e9-kube-api-access-t5plx\") pod \"authorino-6bf9c7b78-bb5jp\" (UID: \"39eb76b8-26c0-41c7-ad67-6918247344e9\") " pod="kuadrant-system/authorino-6bf9c7b78-bb5jp"
Sep 10 17:35:52.334407 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:52.334378    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bf9c7b78-bb5jp"
Sep 10 17:35:52.608699 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:52.608619    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bf9c7b78-bb5jp"]
Sep 10 17:35:52.612202 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:35:52.612166    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39eb76b8_26c0_41c7_ad67_6918247344e9.slice/crio-b62f6cc9bbbcca96b82efb8bce5660df208184dc0dfa6acf96098ca964989181 WatchSource:0}: Error finding container b62f6cc9bbbcca96b82efb8bce5660df208184dc0dfa6acf96098ca964989181: Status 404 returned error can't find the container with id b62f6cc9bbbcca96b82efb8bce5660df208184dc0dfa6acf96098ca964989181
Sep 10 17:35:53.258005 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:53.257974    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bf9c7b78-bb5jp" event={"ID":"39eb76b8-26c0-41c7-ad67-6918247344e9","Type":"ContainerStarted","Data":"b62f6cc9bbbcca96b82efb8bce5660df208184dc0dfa6acf96098ca964989181"}
Sep 10 17:35:54.265648 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:54.265609    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bf9c7b78-bb5jp" event={"ID":"39eb76b8-26c0-41c7-ad67-6918247344e9","Type":"ContainerStarted","Data":"bcb08ebbb3e662fbe0a48e21f5b4fb2c2b2312ff7f6f3602ca5af50fac6a3b40"}
Sep 10 17:35:54.283305 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:35:54.283234    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6bf9c7b78-bb5jp" podStartSLOduration=1.578090816 podStartE2EDuration="2.283204869s" totalImagesPullingTime="705.114053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:35:54.281740399 +0000 UTC m=+726.040267469" watchObservedRunningTime="2026-09-10 17:35:54.283204869 +0000 UTC m=+726.041731935"
Sep 10 17:36:16.001600 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:16.001563    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:36:31.225001 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:31.224968    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:36:40.267873 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.267833    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"]
Sep 10 17:36:40.271488 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.271467    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.274211 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.274185    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 17:36:40.274342 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.274300    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2f1093-openshift-default-dockercfg-7tlll\""
Sep 10 17:36:40.282581 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.282550    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"]
Sep 10 17:36:40.323532 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.323473    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-credential-socket\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.323709 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.323536    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4grg\" (UniqueName: \"kubernetes.io/projected/71c2f787-a899-4312-b5c4-25a22cb787ae-kube-api-access-m4grg\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.323709 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.323580    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-workload-certs\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.323709 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.323599    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-data\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.323709 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.323634    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-podinfo\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.323853 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.323703    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-workload-socket\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.323853 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.323738    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-token\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.323853 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.323781    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-envoy\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.323853 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.323796    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c2f787-a899-4312-b5c4-25a22cb787ae-istiod-ca-cert\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.424356 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.424318    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-credential-socket\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.424356 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.424359    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4grg\" (UniqueName: \"kubernetes.io/projected/71c2f787-a899-4312-b5c4-25a22cb787ae-kube-api-access-m4grg\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.424657 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.424387    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-workload-certs\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.424657 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.424403    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-data\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.424657 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.424437    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-podinfo\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.424657 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.424479    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-workload-socket\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.424657 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.424531    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-token\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.424657 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.424582    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-envoy\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.424657 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.424597    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c2f787-a899-4312-b5c4-25a22cb787ae-istiod-ca-cert\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.425071 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.424956    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-credential-socket\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.425071 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.424984    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-workload-certs\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.425180 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.425140    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-workload-socket\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.425180 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.425170    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-data\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.425474 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.425454    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c2f787-a899-4312-b5c4-25a22cb787ae-istiod-ca-cert\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.427061 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.427036    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-podinfo\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.427231 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.427212    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-envoy\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.432805 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.432780    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-token\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.433087 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.433067    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4grg\" (UniqueName: \"kubernetes.io/projected/71c2f787-a899-4312-b5c4-25a22cb787ae-kube-api-access-m4grg\") pod \"e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") " pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.582628 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.582538    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:40.862445 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:40.862408    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"]
Sep 10 17:36:40.864849 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:36:40.864822    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c2f787_a899_4312_b5c4_25a22cb787ae.slice/crio-9eb61045a9f23d70e4f43583536c38b2d7d2ad9d41890ee2f443f64e79f74071 WatchSource:0}: Error finding container 9eb61045a9f23d70e4f43583536c38b2d7d2ad9d41890ee2f443f64e79f74071: Status 404 returned error can't find the container with id 9eb61045a9f23d70e4f43583536c38b2d7d2ad9d41890ee2f443f64e79f74071
Sep 10 17:36:41.490176 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:41.490141    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb" event={"ID":"71c2f787-a899-4312-b5c4-25a22cb787ae","Type":"ContainerStarted","Data":"9eb61045a9f23d70e4f43583536c38b2d7d2ad9d41890ee2f443f64e79f74071"}
Sep 10 17:36:43.563421 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:43.563382    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:36:43.563800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:43.563474    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:36:43.563800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:43.563550    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:36:44.506514 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:44.506452    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb" event={"ID":"71c2f787-a899-4312-b5c4-25a22cb787ae","Type":"ContainerStarted","Data":"b12725a5050fc49da165e716b810f34cfbfb898934c206378492e8c0719e4523"}
Sep 10 17:36:44.531666 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:44.531612    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb" podStartSLOduration=1.835460203 podStartE2EDuration="4.531595213s" totalImagesPullingTime="2.69613501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:44.530086955 +0000 UTC m=+776.288614026" watchObservedRunningTime="2026-09-10 17:36:44.531595213 +0000 UTC m=+776.290122283"
Sep 10 17:36:44.583510 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:44.583474    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:44.588311 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:44.588286    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:44.984292 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:44.984257    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"]
Sep 10 17:36:44.987721 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:44.987705    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:44.990622 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:44.990600    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-21724959-kserve-self-signed-certs\""
Sep 10 17:36:44.997077 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:44.997056    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"]
Sep 10 17:36:45.070600 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.070562    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-dshm\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.070776 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.070616    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g4r2\" (UniqueName: \"kubernetes.io/projected/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-kube-api-access-7g4r2\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.070776 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.070694    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-tls-certs\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.070776 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.070727    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-tmp-dir\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.070776 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.070751    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-home\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.070776 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.070769    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-model-cache\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.171370 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.171337    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-dshm\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.171549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.171388    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g4r2\" (UniqueName: \"kubernetes.io/projected/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-kube-api-access-7g4r2\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.171549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.171424    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-tls-certs\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.171549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.171442    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-tmp-dir\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.171549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.171460    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-home\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.171549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.171481    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-model-cache\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.171905 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.171879    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-tmp-dir\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.171982 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.171903    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-home\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.171982 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.171928    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-model-cache\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.173578 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.173551    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-dshm\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.173848 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.173829    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-tls-certs\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.182737 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.182718    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g4r2\" (UniqueName: \"kubernetes.io/projected/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-kube-api-access-7g4r2\") pod \"e2e-conflict-a-21724959-kserve-776c8498db-bdgvp\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") " pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.297653 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.297544    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:45.512455 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.512425    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:45.513282 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.513261    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:36:45.574621 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:45.574592    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"]
Sep 10 17:36:45.576694 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:36:45.576664    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e16b75c_cffd_4af2_af38_ccdf202dc9fa.slice/crio-78771d83f0a458e9986f3398d4352dd81d565c1959f073c608bf0d2c3dc7889c WatchSource:0}: Error finding container 78771d83f0a458e9986f3398d4352dd81d565c1959f073c608bf0d2c3dc7889c: Status 404 returned error can't find the container with id 78771d83f0a458e9986f3398d4352dd81d565c1959f073c608bf0d2c3dc7889c
Sep 10 17:36:46.519553 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:46.519474    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp" event={"ID":"4e16b75c-cffd-4af2-af38-ccdf202dc9fa","Type":"ContainerStarted","Data":"ebf027d0c41b4c3745994f0c81aec55c5024b3af3296071aa3d7f9bdd7042c70"}
Sep 10 17:36:46.519553 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:46.519546    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp" event={"ID":"4e16b75c-cffd-4af2-af38-ccdf202dc9fa","Type":"ContainerStarted","Data":"78771d83f0a458e9986f3398d4352dd81d565c1959f073c608bf0d2c3dc7889c"}
Sep 10 17:36:46.544263 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:46.544210    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp" podStartSLOduration=2.5441950110000002 podStartE2EDuration="2.544195011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:46.541966697 +0000 UTC m=+778.300493768" watchObservedRunningTime="2026-09-10 17:36:46.544195011 +0000 UTC m=+778.302722082"
Sep 10 17:36:48.803645 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:48.803617    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58"]
Sep 10 17:36:48.807229 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:48.807203    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58"
Sep 10 17:36:48.810100 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:48.810063    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58"]
Sep 10 17:36:48.811111 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:48.811086    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 17:36:48.812271 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:48.812246    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:36:48.908624 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:48.908584    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdce8b22-b85a-4fa4-9707-43def1a81c24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58\" (UID: \"bdce8b22-b85a-4fa4-9707-43def1a81c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58"
Sep 10 17:36:49.009323 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:49.009280    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdce8b22-b85a-4fa4-9707-43def1a81c24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58\" (UID: \"bdce8b22-b85a-4fa4-9707-43def1a81c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58"
Sep 10 17:36:49.009514 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:49.009295    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:36:49.010186 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:49.010158    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdce8b22-b85a-4fa4-9707-43def1a81c24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58\" (UID: \"bdce8b22-b85a-4fa4-9707-43def1a81c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58"
Sep 10 17:36:49.119899 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:49.119870    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58"
Sep 10 17:36:49.401884 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:49.401858    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58"]
Sep 10 17:36:49.403988 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:36:49.403960    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdce8b22_b85a_4fa4_9707_43def1a81c24.slice/crio-b01760e81861a13fbfd2a10acaff98574d1573ef6c3add4dc9069159f0544811 WatchSource:0}: Error finding container b01760e81861a13fbfd2a10acaff98574d1573ef6c3add4dc9069159f0544811: Status 404 returned error can't find the container with id b01760e81861a13fbfd2a10acaff98574d1573ef6c3add4dc9069159f0544811
Sep 10 17:36:49.534813 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:49.534778    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58" event={"ID":"bdce8b22-b85a-4fa4-9707-43def1a81c24","Type":"ContainerStarted","Data":"b01760e81861a13fbfd2a10acaff98574d1573ef6c3add4dc9069159f0544811"}
Sep 10 17:36:52.554741 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:52.554700    2552 generic.go:356] "Generic (PLEG): container finished" podID="bdce8b22-b85a-4fa4-9707-43def1a81c24" containerID="f6281279c86a18dc37cbfe3f642bc53292857a5d0f6d81c25a07f0ae448baf36" exitCode=0
Sep 10 17:36:52.555143 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:52.554782    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58" event={"ID":"bdce8b22-b85a-4fa4-9707-43def1a81c24","Type":"ContainerDied","Data":"f6281279c86a18dc37cbfe3f642bc53292857a5d0f6d81c25a07f0ae448baf36"}
Sep 10 17:36:53.689123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:53.689092    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58"
Sep 10 17:36:53.757050 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:53.757005    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdce8b22-b85a-4fa4-9707-43def1a81c24-maas-api-service-ca\") pod \"bdce8b22-b85a-4fa4-9707-43def1a81c24\" (UID: \"bdce8b22-b85a-4fa4-9707-43def1a81c24\") "
Sep 10 17:36:53.757374 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:53.757352    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdce8b22-b85a-4fa4-9707-43def1a81c24-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bdce8b22-b85a-4fa4-9707-43def1a81c24" (UID: "bdce8b22-b85a-4fa4-9707-43def1a81c24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:36:53.858073 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:53.858038    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdce8b22-b85a-4fa4-9707-43def1a81c24-maas-api-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:36:54.565172 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:54.565141    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58"
Sep 10 17:36:54.565363 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:54.565140    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58" event={"ID":"bdce8b22-b85a-4fa4-9707-43def1a81c24","Type":"ContainerDied","Data":"b01760e81861a13fbfd2a10acaff98574d1573ef6c3add4dc9069159f0544811"}
Sep 10 17:36:54.565363 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:54.565284    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b01760e81861a13fbfd2a10acaff98574d1573ef6c3add4dc9069159f0544811"
Sep 10 17:36:55.297779 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:55.297743    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:55.298201 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:55.297831    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:55.310249 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:55.310222    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:36:55.581288 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:36:55.581206    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:37:03.005205 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:03.005166    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:37:05.757209 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:05.757165    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58"]
Sep 10 17:37:05.760003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:05.759976    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64df4d9a-6hp58"]
Sep 10 17:37:06.803484 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.803453    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdce8b22-b85a-4fa4-9707-43def1a81c24" path="/var/lib/kubelet/pods/bdce8b22-b85a-4fa4-9707-43def1a81c24/volumes"
Sep 10 17:37:06.877961 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.877929    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"]
Sep 10 17:37:06.878385 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.878366    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdce8b22-b85a-4fa4-9707-43def1a81c24" containerName="revoke-keys"
Sep 10 17:37:06.878480 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.878387    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdce8b22-b85a-4fa4-9707-43def1a81c24" containerName="revoke-keys"
Sep 10 17:37:06.878574 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.878550    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdce8b22-b85a-4fa4-9707-43def1a81c24" containerName="revoke-keys"
Sep 10 17:37:06.880803 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.880768    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:06.884314 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.884285    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-fb3998b8-gw-openshift-default-dockercfg-n7l6c\""
Sep 10 17:37:06.896190 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.896165    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"]
Sep 10 17:37:06.987077 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.987038    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-workload-socket\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:06.987222 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.987090    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-istio-data\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:06.987222 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.987154    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-workload-certs\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:06.987222 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.987206    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-istio-envoy\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:06.987332 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.987232    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68q2n\" (UniqueName: \"kubernetes.io/projected/2fcf0801-66b2-4814-b031-b004304541fc-kube-api-access-68q2n\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:06.987332 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.987266    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fcf0801-66b2-4814-b031-b004304541fc-istiod-ca-cert\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:06.987332 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.987283    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-credential-socket\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:06.987332 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.987302    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fcf0801-66b2-4814-b031-b004304541fc-istio-token\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:06.987555 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:06.987413    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fcf0801-66b2-4814-b031-b004304541fc-istio-podinfo\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.088805 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.088771    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-workload-certs\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.088963 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.088811    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-istio-envoy\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.088963 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.088836    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68q2n\" (UniqueName: \"kubernetes.io/projected/2fcf0801-66b2-4814-b031-b004304541fc-kube-api-access-68q2n\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.088963 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.088878    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fcf0801-66b2-4814-b031-b004304541fc-istiod-ca-cert\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.088963 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.088902    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-credential-socket\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.088963 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.088926    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fcf0801-66b2-4814-b031-b004304541fc-istio-token\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.089243 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.088978    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fcf0801-66b2-4814-b031-b004304541fc-istio-podinfo\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.089243 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.089071    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-workload-socket\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.089243 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.089127    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-istio-data\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.089243 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.089223    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-workload-certs\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.089556 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.089532    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fcf0801-66b2-4814-b031-b004304541fc-istiod-ca-cert\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.089806 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.089770    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-istio-data\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.089806 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.089793    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-credential-socket\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.089957 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.089823    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-workload-socket\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.091148 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.091125    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-istio-envoy\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.091673 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.091642    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fcf0801-66b2-4814-b031-b004304541fc-istio-podinfo\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.098744 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.098713    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fcf0801-66b2-4814-b031-b004304541fc-istio-token\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.099248 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.099217    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68q2n\" (UniqueName: \"kubernetes.io/projected/2fcf0801-66b2-4814-b031-b004304541fc-kube-api-access-68q2n\") pod \"e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") " pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.251980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.251939    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:07.533132 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.533104    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"]
Sep 10 17:37:07.534864 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:37:07.534840    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcf0801_66b2_4814_b031_b004304541fc.slice/crio-302d5e77e9c76e41c42af483b14dae1c11a3b33b89f390394aad727eab52e821 WatchSource:0}: Error finding container 302d5e77e9c76e41c42af483b14dae1c11a3b33b89f390394aad727eab52e821: Status 404 returned error can't find the container with id 302d5e77e9c76e41c42af483b14dae1c11a3b33b89f390394aad727eab52e821
Sep 10 17:37:07.537167 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.537135    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:37:07.537232 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.537215    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:37:07.537274 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.537247    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:37:07.639850 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.639811    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs" event={"ID":"2fcf0801-66b2-4814-b031-b004304541fc","Type":"ContainerStarted","Data":"2a310640cf9a8be7574520ed3ffd8523832d8fc6c96001397f6ac2d70cc94b55"}
Sep 10 17:37:07.640144 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.639912    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs" event={"ID":"2fcf0801-66b2-4814-b031-b004304541fc","Type":"ContainerStarted","Data":"302d5e77e9c76e41c42af483b14dae1c11a3b33b89f390394aad727eab52e821"}
Sep 10 17:37:07.642091 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.642070    2552 generic.go:356] "Generic (PLEG): container finished" podID="71c2f787-a899-4312-b5c4-25a22cb787ae" containerID="b12725a5050fc49da165e716b810f34cfbfb898934c206378492e8c0719e4523" exitCode=0
Sep 10 17:37:07.642204 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.642141    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb" event={"ID":"71c2f787-a899-4312-b5c4-25a22cb787ae","Type":"ContainerDied","Data":"b12725a5050fc49da165e716b810f34cfbfb898934c206378492e8c0719e4523"}
Sep 10 17:37:07.642542 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.642528    2552 scope.go:117] "RemoveContainer" containerID="b12725a5050fc49da165e716b810f34cfbfb898934c206378492e8c0719e4523"
Sep 10 17:37:07.643675 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.643640    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:37:07.643788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.643727    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:37:07.643788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.643774    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:37:07.665075 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:07.665019    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs" podStartSLOduration=1.664999914 podStartE2EDuration="1.664999914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:07.663272235 +0000 UTC m=+799.421799308" watchObservedRunningTime="2026-09-10 17:37:07.664999914 +0000 UTC m=+799.423526985"
Sep 10 17:37:08.252454 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:08.252422    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:08.257260 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:08.257235    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:08.651802 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:08.651758    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb" event={"ID":"71c2f787-a899-4312-b5c4-25a22cb787ae","Type":"ContainerStarted","Data":"d0ce2e14922f1e3880a31e13ca1355f29566969b48d56b697ce108ff2cbe8454"}
Sep 10 17:37:08.653092 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:08.653064    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:08.654821 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:08.654761    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:09.582940 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:09.582903    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:37:09.583418 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:09.583158    2552 patch_prober.go:28] interesting pod/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 10 17:37:09.583418 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:09.583242    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb" podUID="71c2f787-a899-4312-b5c4-25a22cb787ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 10 17:37:10.583227 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:10.583192    2552 patch_prober.go:28] interesting pod/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 10 17:37:10.583660 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:10.583257    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb" podUID="71c2f787-a899-4312-b5c4-25a22cb787ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 10 17:37:11.583387 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:11.583306    2552 patch_prober.go:28] interesting pod/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 10 17:37:11.583791 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:11.583377    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb" podUID="71c2f787-a899-4312-b5c4-25a22cb787ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 10 17:37:12.583693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:12.583659    2552 patch_prober.go:28] interesting pod/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 10 17:37:12.584097 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:12.583721    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb" podUID="71c2f787-a899-4312-b5c4-25a22cb787ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 10 17:37:13.583143 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:13.583107    2552 patch_prober.go:28] interesting pod/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 10 17:37:13.583322 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:13.583164    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb" podUID="71c2f787-a899-4312-b5c4-25a22cb787ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 10 17:37:14.587392 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:14.587351    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:37:14.587789 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:14.587709    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:37:14.588437 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:14.588421    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:37:19.006269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:19.006229    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:37:30.769806 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:30.769770    2552 generic.go:356] "Generic (PLEG): container finished" podID="2fcf0801-66b2-4814-b031-b004304541fc" containerID="2a310640cf9a8be7574520ed3ffd8523832d8fc6c96001397f6ac2d70cc94b55" exitCode=0
Sep 10 17:37:30.770221 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:30.769826    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs" event={"ID":"2fcf0801-66b2-4814-b031-b004304541fc","Type":"ContainerDied","Data":"2a310640cf9a8be7574520ed3ffd8523832d8fc6c96001397f6ac2d70cc94b55"}
Sep 10 17:37:30.770280 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:30.770265    2552 scope.go:117] "RemoveContainer" containerID="2a310640cf9a8be7574520ed3ffd8523832d8fc6c96001397f6ac2d70cc94b55"
Sep 10 17:37:30.771343 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:30.771314    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:37:30.771395 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:30.771382    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:37:30.771431 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:30.771417    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:37:31.777732 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:31.777693    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs" event={"ID":"2fcf0801-66b2-4814-b031-b004304541fc","Type":"ContainerStarted","Data":"748178764b8b801c958a5908ebe501d83a9e6a87657b374f5c35c59b9b3b942f"}
Sep 10 17:37:32.252345 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:32.252300    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:32.252631 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:32.252605    2552 patch_prober.go:28] interesting pod/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 10 17:37:32.252739 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:32.252663    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs" podUID="2fcf0801-66b2-4814-b031-b004304541fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 10 17:37:32.805059 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:32.805020    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:37:33.256395 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:33.255943    2552 patch_prober.go:28] interesting pod/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 10 17:37:33.256395 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:33.256008    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs" podUID="2fcf0801-66b2-4814-b031-b004304541fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 10 17:37:34.094091 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:34.094057    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd"]
Sep 10 17:37:34.096891 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:34.096865    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd"
Sep 10 17:37:34.099439 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:34.099414    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:37:34.099439 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:34.099414    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 17:37:34.104908 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:34.104885    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd"]
Sep 10 17:37:34.176763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:34.176714    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/736bfb23-d8c0-4eca-b148-c2a2387e8f7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd\" (UID: \"736bfb23-d8c0-4eca-b148-c2a2387e8f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd"
Sep 10 17:37:34.252900 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:34.252849    2552 patch_prober.go:28] interesting pod/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 10 17:37:34.253123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:34.252925    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs" podUID="2fcf0801-66b2-4814-b031-b004304541fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 10 17:37:34.278264 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:34.278222    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/736bfb23-d8c0-4eca-b148-c2a2387e8f7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd\" (UID: \"736bfb23-d8c0-4eca-b148-c2a2387e8f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd"
Sep 10 17:37:34.278950 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:34.278926    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/736bfb23-d8c0-4eca-b148-c2a2387e8f7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd\" (UID: \"736bfb23-d8c0-4eca-b148-c2a2387e8f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd"
Sep 10 17:37:34.408293 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:34.408206    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd"
Sep 10 17:37:34.703905 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:34.703880    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd"]
Sep 10 17:37:34.705947 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:37:34.705917    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736bfb23_d8c0_4eca_b148_c2a2387e8f7f.slice/crio-f395d9daf04155b9d70a6d928f1416f04a81f1cf9ed7d151267ab781570f0d3d WatchSource:0}: Error finding container f395d9daf04155b9d70a6d928f1416f04a81f1cf9ed7d151267ab781570f0d3d: Status 404 returned error can't find the container with id f395d9daf04155b9d70a6d928f1416f04a81f1cf9ed7d151267ab781570f0d3d
Sep 10 17:37:34.802908 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:34.802870    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd" event={"ID":"736bfb23-d8c0-4eca-b148-c2a2387e8f7f","Type":"ContainerStarted","Data":"f395d9daf04155b9d70a6d928f1416f04a81f1cf9ed7d151267ab781570f0d3d"}
Sep 10 17:37:35.252558 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.252434    2552 patch_prober.go:28] interesting pod/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 10 17:37:35.252558 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.252534    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs" podUID="2fcf0801-66b2-4814-b031-b004304541fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 10 17:37:35.799208 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.799175    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"]
Sep 10 17:37:35.802214 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.802191    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.805274 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.805249    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-21724959-kserve-self-signed-certs\""
Sep 10 17:37:35.807296 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.807275    2552 generic.go:356] "Generic (PLEG): container finished" podID="736bfb23-d8c0-4eca-b148-c2a2387e8f7f" containerID="0bcb6d2a0d837fd63c4076151f812448c703aa21750880ac686091a118d239ab" exitCode=0
Sep 10 17:37:35.807400 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.807351    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd" event={"ID":"736bfb23-d8c0-4eca-b148-c2a2387e8f7f","Type":"ContainerDied","Data":"0bcb6d2a0d837fd63c4076151f812448c703aa21750880ac686091a118d239ab"}
Sep 10 17:37:35.813926 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.813902    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"]
Sep 10 17:37:35.893173 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.893139    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-tmp-dir\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.893387 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.893214    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-model-cache\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.893387 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.893266    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-home\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.893387 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.893355    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-dshm\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.893387 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.893381    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b6924-e415-4afc-9298-31c8aaf5f600-tls-certs\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.893628 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.893406    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skxxb\" (UniqueName: \"kubernetes.io/projected/1f3b6924-e415-4afc-9298-31c8aaf5f600-kube-api-access-skxxb\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.994731 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.994634    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-model-cache\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.994731 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.994677    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-home\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.994731 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.994722    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-dshm\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.995011 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.994738    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b6924-e415-4afc-9298-31c8aaf5f600-tls-certs\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.995011 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.994755    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skxxb\" (UniqueName: \"kubernetes.io/projected/1f3b6924-e415-4afc-9298-31c8aaf5f600-kube-api-access-skxxb\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.995011 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.994809    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-tmp-dir\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.995174 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.995119    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-model-cache\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.995264 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.995239    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-home\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.995316 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.995298    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-tmp-dir\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.997114 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.997093    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-dshm\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:35.997304 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:35.997287    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b6924-e415-4afc-9298-31c8aaf5f600-tls-certs\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:36.002875 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:36.002851    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skxxb\" (UniqueName: \"kubernetes.io/projected/1f3b6924-e415-4afc-9298-31c8aaf5f600-kube-api-access-skxxb\") pod \"e2e-conflict-b-21724959-kserve-69868dc9b-p7d25\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") " pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:36.113453 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:36.113423    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:36.269008 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:36.268903    2552 patch_prober.go:28] interesting pod/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 10 17:37:36.269008 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:36.268989    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs" podUID="2fcf0801-66b2-4814-b031-b004304541fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 10 17:37:36.407858 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:36.407821    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"]
Sep 10 17:37:36.410301 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:37:36.410264    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f3b6924_e415_4afc_9298_31c8aaf5f600.slice/crio-87a463d420424d0dd34d09a8e9c62292faec78b29e078a993bb799f1bcd74f1d WatchSource:0}: Error finding container 87a463d420424d0dd34d09a8e9c62292faec78b29e078a993bb799f1bcd74f1d: Status 404 returned error can't find the container with id 87a463d420424d0dd34d09a8e9c62292faec78b29e078a993bb799f1bcd74f1d
Sep 10 17:37:36.817552 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:36.817431    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25" event={"ID":"1f3b6924-e415-4afc-9298-31c8aaf5f600","Type":"ContainerStarted","Data":"c0147932453ca562716fd6d95c50a975de37db160d96427ea97c89a3655f6936"}
Sep 10 17:37:36.817552 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:36.817474    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25" event={"ID":"1f3b6924-e415-4afc-9298-31c8aaf5f600","Type":"ContainerStarted","Data":"87a463d420424d0dd34d09a8e9c62292faec78b29e078a993bb799f1bcd74f1d"}
Sep 10 17:37:36.836629 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:36.836578    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25" podStartSLOduration=1.836562232 podStartE2EDuration="1.836562232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:36.835394124 +0000 UTC m=+828.593921193" watchObservedRunningTime="2026-09-10 17:37:36.836562232 +0000 UTC m=+828.595089302"
Sep 10 17:37:36.992371 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:36.992345    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd"
Sep 10 17:37:37.104966 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:37.104932    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/736bfb23-d8c0-4eca-b148-c2a2387e8f7f-maas-api-service-ca\") pod \"736bfb23-d8c0-4eca-b148-c2a2387e8f7f\" (UID: \"736bfb23-d8c0-4eca-b148-c2a2387e8f7f\") "
Sep 10 17:37:37.105281 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:37.105257    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736bfb23-d8c0-4eca-b148-c2a2387e8f7f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "736bfb23-d8c0-4eca-b148-c2a2387e8f7f" (UID: "736bfb23-d8c0-4eca-b148-c2a2387e8f7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:37:37.206651 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:37.206616    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/736bfb23-d8c0-4eca-b148-c2a2387e8f7f-maas-api-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:37:37.252831 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:37.252780    2552 patch_prober.go:28] interesting pod/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 10 17:37:37.253045 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:37.252854    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs" podUID="2fcf0801-66b2-4814-b031-b004304541fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 10 17:37:37.823875 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:37.823843    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd"
Sep 10 17:37:37.823875 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:37.823865    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd" event={"ID":"736bfb23-d8c0-4eca-b148-c2a2387e8f7f","Type":"ContainerDied","Data":"f395d9daf04155b9d70a6d928f1416f04a81f1cf9ed7d151267ab781570f0d3d"}
Sep 10 17:37:37.824344 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:37.823908    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f395d9daf04155b9d70a6d928f1416f04a81f1cf9ed7d151267ab781570f0d3d"
Sep 10 17:37:38.252632 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:38.252593    2552 patch_prober.go:28] interesting pod/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 10 17:37:38.252831 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:38.252681    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs" podUID="2fcf0801-66b2-4814-b031-b004304541fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 10 17:37:38.653420 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:38.653385    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:39.256944 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:39.256913    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:39.257896 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:39.257876    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:40.907067 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:40.907030    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd"]
Sep 10 17:37:40.909622 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:40.909598    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb3998b8-qf9xd"]
Sep 10 17:37:41.038881 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:41.038845    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"]
Sep 10 17:37:41.039103 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:41.039071    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs" podUID="2fcf0801-66b2-4814-b031-b004304541fc" containerName="istio-proxy" containerID="cri-o://748178764b8b801c958a5908ebe501d83a9e6a87657b374f5c35c59b9b3b942f" gracePeriod=30
Sep 10 17:37:42.803701 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:42.803657    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="736bfb23-d8c0-4eca-b148-c2a2387e8f7f" path="/var/lib/kubelet/pods/736bfb23-d8c0-4eca-b148-c2a2387e8f7f/volumes"
Sep 10 17:37:46.113866 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.113807    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:46.114248 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.113890    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:46.126915 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.126883    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:46.386640 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.386613    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:46.504485 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.504449    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fcf0801-66b2-4814-b031-b004304541fc-istiod-ca-cert\") pod \"2fcf0801-66b2-4814-b031-b004304541fc\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") "
Sep 10 17:37:46.504660 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.504552    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68q2n\" (UniqueName: \"kubernetes.io/projected/2fcf0801-66b2-4814-b031-b004304541fc-kube-api-access-68q2n\") pod \"2fcf0801-66b2-4814-b031-b004304541fc\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") "
Sep 10 17:37:46.504660 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.504594    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-workload-certs\") pod \"2fcf0801-66b2-4814-b031-b004304541fc\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") "
Sep 10 17:37:46.504761 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.504670    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fcf0801-66b2-4814-b031-b004304541fc-istio-podinfo\") pod \"2fcf0801-66b2-4814-b031-b004304541fc\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") "
Sep 10 17:37:46.504761 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.504700    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fcf0801-66b2-4814-b031-b004304541fc-istio-token\") pod \"2fcf0801-66b2-4814-b031-b004304541fc\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") "
Sep 10 17:37:46.504761 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.504737    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-istio-data\") pod \"2fcf0801-66b2-4814-b031-b004304541fc\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") "
Sep 10 17:37:46.504917 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.504776    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-credential-socket\") pod \"2fcf0801-66b2-4814-b031-b004304541fc\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") "
Sep 10 17:37:46.504917 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.504846    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-workload-socket\") pod \"2fcf0801-66b2-4814-b031-b004304541fc\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") "
Sep 10 17:37:46.504917 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.504859    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fcf0801-66b2-4814-b031-b004304541fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2fcf0801-66b2-4814-b031-b004304541fc" (UID: "2fcf0801-66b2-4814-b031-b004304541fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:37:46.505084 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.504964    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2fcf0801-66b2-4814-b031-b004304541fc" (UID: "2fcf0801-66b2-4814-b031-b004304541fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:37:46.505084 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.504882    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-istio-envoy\") pod \"2fcf0801-66b2-4814-b031-b004304541fc\" (UID: \"2fcf0801-66b2-4814-b031-b004304541fc\") "
Sep 10 17:37:46.505467 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.505188    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "2fcf0801-66b2-4814-b031-b004304541fc" (UID: "2fcf0801-66b2-4814-b031-b004304541fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:37:46.505467 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.505382    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2fcf0801-66b2-4814-b031-b004304541fc" (UID: "2fcf0801-66b2-4814-b031-b004304541fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:37:46.505467 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.505404    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2fcf0801-66b2-4814-b031-b004304541fc" (UID: "2fcf0801-66b2-4814-b031-b004304541fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:37:46.505467 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.505423    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fcf0801-66b2-4814-b031-b004304541fc-istiod-ca-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:37:46.505467 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.505444    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-workload-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:37:46.505467 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.505457    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-istio-data\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:37:46.507329 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.507301    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fcf0801-66b2-4814-b031-b004304541fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "2fcf0801-66b2-4814-b031-b004304541fc" (UID: "2fcf0801-66b2-4814-b031-b004304541fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:37:46.507542 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.507487    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2fcf0801-66b2-4814-b031-b004304541fc" (UID: "2fcf0801-66b2-4814-b031-b004304541fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:37:46.507639 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.507526    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fcf0801-66b2-4814-b031-b004304541fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2fcf0801-66b2-4814-b031-b004304541fc" (UID: "2fcf0801-66b2-4814-b031-b004304541fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:37:46.507639 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.507592    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fcf0801-66b2-4814-b031-b004304541fc-kube-api-access-68q2n" (OuterVolumeSpecName: "kube-api-access-68q2n") pod "2fcf0801-66b2-4814-b031-b004304541fc" (UID: "2fcf0801-66b2-4814-b031-b004304541fc"). InnerVolumeSpecName "kube-api-access-68q2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:37:46.606139 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.606108    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-workload-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:37:46.606139 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.606138    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-istio-envoy\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:37:46.606369 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.606153    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68q2n\" (UniqueName: \"kubernetes.io/projected/2fcf0801-66b2-4814-b031-b004304541fc-kube-api-access-68q2n\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:37:46.606369 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.606166    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fcf0801-66b2-4814-b031-b004304541fc-istio-podinfo\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:37:46.606369 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.606191    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fcf0801-66b2-4814-b031-b004304541fc-istio-token\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:37:46.606369 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.606201    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fcf0801-66b2-4814-b031-b004304541fc-credential-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:37:46.871288 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.871251    2552 generic.go:356] "Generic (PLEG): container finished" podID="2fcf0801-66b2-4814-b031-b004304541fc" containerID="748178764b8b801c958a5908ebe501d83a9e6a87657b374f5c35c59b9b3b942f" exitCode=0
Sep 10 17:37:46.871461 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.871304    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"
Sep 10 17:37:46.871461 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.871343    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs" event={"ID":"2fcf0801-66b2-4814-b031-b004304541fc","Type":"ContainerDied","Data":"748178764b8b801c958a5908ebe501d83a9e6a87657b374f5c35c59b9b3b942f"}
Sep 10 17:37:46.871461 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.871405    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs" event={"ID":"2fcf0801-66b2-4814-b031-b004304541fc","Type":"ContainerDied","Data":"302d5e77e9c76e41c42af483b14dae1c11a3b33b89f390394aad727eab52e821"}
Sep 10 17:37:46.871461 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.871433    2552 scope.go:117] "RemoveContainer" containerID="748178764b8b801c958a5908ebe501d83a9e6a87657b374f5c35c59b9b3b942f"
Sep 10 17:37:46.882364 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.882348    2552 scope.go:117] "RemoveContainer" containerID="2a310640cf9a8be7574520ed3ffd8523832d8fc6c96001397f6ac2d70cc94b55"
Sep 10 17:37:46.886149 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.886126    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:37:46.888699 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.888678    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"]
Sep 10 17:37:46.891599 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.891580    2552 scope.go:117] "RemoveContainer" containerID="748178764b8b801c958a5908ebe501d83a9e6a87657b374f5c35c59b9b3b942f"
Sep 10 17:37:46.891890 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:37:46.891869    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748178764b8b801c958a5908ebe501d83a9e6a87657b374f5c35c59b9b3b942f\": container with ID starting with 748178764b8b801c958a5908ebe501d83a9e6a87657b374f5c35c59b9b3b942f not found: ID does not exist" containerID="748178764b8b801c958a5908ebe501d83a9e6a87657b374f5c35c59b9b3b942f"
Sep 10 17:37:46.891959 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.891899    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748178764b8b801c958a5908ebe501d83a9e6a87657b374f5c35c59b9b3b942f"} err="failed to get container status \"748178764b8b801c958a5908ebe501d83a9e6a87657b374f5c35c59b9b3b942f\": rpc error: code = NotFound desc = could not find container \"748178764b8b801c958a5908ebe501d83a9e6a87657b374f5c35c59b9b3b942f\": container with ID starting with 748178764b8b801c958a5908ebe501d83a9e6a87657b374f5c35c59b9b3b942f not found: ID does not exist"
Sep 10 17:37:46.891959 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.891920    2552 scope.go:117] "RemoveContainer" containerID="2a310640cf9a8be7574520ed3ffd8523832d8fc6c96001397f6ac2d70cc94b55"
Sep 10 17:37:46.892179 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:37:46.892161    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a310640cf9a8be7574520ed3ffd8523832d8fc6c96001397f6ac2d70cc94b55\": container with ID starting with 2a310640cf9a8be7574520ed3ffd8523832d8fc6c96001397f6ac2d70cc94b55 not found: ID does not exist" containerID="2a310640cf9a8be7574520ed3ffd8523832d8fc6c96001397f6ac2d70cc94b55"
Sep 10 17:37:46.892239 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.892186    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a310640cf9a8be7574520ed3ffd8523832d8fc6c96001397f6ac2d70cc94b55"} err="failed to get container status \"2a310640cf9a8be7574520ed3ffd8523832d8fc6c96001397f6ac2d70cc94b55\": rpc error: code = NotFound desc = could not find container \"2a310640cf9a8be7574520ed3ffd8523832d8fc6c96001397f6ac2d70cc94b55\": container with ID starting with 2a310640cf9a8be7574520ed3ffd8523832d8fc6c96001397f6ac2d70cc94b55 not found: ID does not exist"
Sep 10 17:37:46.892744 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:46.892722    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-fb3998b8-gw-openshift-default-74d65bbc76-bbfgs"]
Sep 10 17:37:48.806557 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:48.804210    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fcf0801-66b2-4814-b031-b004304541fc" path="/var/lib/kubelet/pods/2fcf0801-66b2-4814-b031-b004304541fc/volumes"
Sep 10 17:37:51.407987 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:37:51.407948    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:38:06.708069 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.708037    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"]
Sep 10 17:38:06.708523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.708420    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="736bfb23-d8c0-4eca-b148-c2a2387e8f7f" containerName="revoke-keys"
Sep 10 17:38:06.708523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.708432    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="736bfb23-d8c0-4eca-b148-c2a2387e8f7f" containerName="revoke-keys"
Sep 10 17:38:06.708523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.708444    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fcf0801-66b2-4814-b031-b004304541fc" containerName="istio-proxy"
Sep 10 17:38:06.708523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.708449    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fcf0801-66b2-4814-b031-b004304541fc" containerName="istio-proxy"
Sep 10 17:38:06.708523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.708466    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fcf0801-66b2-4814-b031-b004304541fc" containerName="istio-proxy"
Sep 10 17:38:06.708523 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.708471    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fcf0801-66b2-4814-b031-b004304541fc" containerName="istio-proxy"
Sep 10 17:38:06.708727 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.708555    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fcf0801-66b2-4814-b031-b004304541fc" containerName="istio-proxy"
Sep 10 17:38:06.708727 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.708568    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="736bfb23-d8c0-4eca-b148-c2a2387e8f7f" containerName="revoke-keys"
Sep 10 17:38:06.713194 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.713177    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:06.716104 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.716083    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-794622\"/\"default-dockercfg-8fhg6\""
Sep 10 17:38:06.716256 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.716083    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-794622\"/\"kube-root-ca.crt\""
Sep 10 17:38:06.717245 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.717226    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-794622\"/\"autores-model-794622-kserve-self-signed-certs\""
Sep 10 17:38:06.717348 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.717256    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-794622\"/\"openshift-service-ca.crt\""
Sep 10 17:38:06.724665 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.724638    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"]
Sep 10 17:38:06.905736 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.905694    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp2vp\" (UniqueName: \"kubernetes.io/projected/1396e850-ed20-41d3-ac05-b9c7c3275e8d-kube-api-access-hp2vp\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:06.905937 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.905765    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-home\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:06.905937 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.905783    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-model-cache\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:06.906072 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.905986    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-tmp-dir\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:06.906072 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.906023    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1396e850-ed20-41d3-ac05-b9c7c3275e8d-tls-certs\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:06.906154 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:06.906093    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-dshm\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:07.007212 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.007119    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-dshm\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:07.007212 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.007168    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp2vp\" (UniqueName: \"kubernetes.io/projected/1396e850-ed20-41d3-ac05-b9c7c3275e8d-kube-api-access-hp2vp\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:07.007212 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.007202    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-home\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:07.007440 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.007222    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-model-cache\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:07.007440 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.007303    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-tmp-dir\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:07.007440 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.007327    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1396e850-ed20-41d3-ac05-b9c7c3275e8d-tls-certs\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:07.007689 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.007664    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-home\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:07.007742 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.007685    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-tmp-dir\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:07.007742 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.007733    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-model-cache\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:07.009470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.009444    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-dshm\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:07.009740 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.009722    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1396e850-ed20-41d3-ac05-b9c7c3275e8d-tls-certs\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:07.015101 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.015070    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp2vp\" (UniqueName: \"kubernetes.io/projected/1396e850-ed20-41d3-ac05-b9c7c3275e8d-kube-api-access-hp2vp\") pod \"autores-model-794622-kserve-8b589975f-l9hvj\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") " pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:07.022381 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.022363    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:07.292260 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.292233    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"]
Sep 10 17:38:07.294192 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:38:07.294164    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1396e850_ed20_41d3_ac05_b9c7c3275e8d.slice/crio-b5e7545c32cf5bba00c45070c16ecba4d51903c189e0e3e70f9624d14bbae18b WatchSource:0}: Error finding container b5e7545c32cf5bba00c45070c16ecba4d51903c189e0e3e70f9624d14bbae18b: Status 404 returned error can't find the container with id b5e7545c32cf5bba00c45070c16ecba4d51903c189e0e3e70f9624d14bbae18b
Sep 10 17:38:07.980892 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.980853    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj" event={"ID":"1396e850-ed20-41d3-ac05-b9c7c3275e8d","Type":"ContainerStarted","Data":"48687698977e3a013dbedc86782cb992949d3b6df435e7acc4e1663b008f4f2d"}
Sep 10 17:38:07.981303 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:07.980904    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj" event={"ID":"1396e850-ed20-41d3-ac05-b9c7c3275e8d","Type":"ContainerStarted","Data":"b5e7545c32cf5bba00c45070c16ecba4d51903c189e0e3e70f9624d14bbae18b"}
Sep 10 17:38:08.000604 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:08.000552    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj" podStartSLOduration=2.000536634 podStartE2EDuration="2.000536634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:07.999483298 +0000 UTC m=+859.758010369" watchObservedRunningTime="2026-09-10 17:38:08.000536634 +0000 UTC m=+859.759063703"
Sep 10 17:38:08.794278 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:08.794247    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:38:10.467779 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:10.467738    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz"]
Sep 10 17:38:10.468182 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:10.468169    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fcf0801-66b2-4814-b031-b004304541fc" containerName="istio-proxy"
Sep 10 17:38:10.472405 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:10.472368    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz"
Sep 10 17:38:10.477612 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:10.477581    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 17:38:10.477801 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:10.477686    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:38:10.480632 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:10.480606    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz"]
Sep 10 17:38:10.534316 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:10.534279    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa6f035-6d5b-4aa4-94fc-01385154ce58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz\" (UID: \"7fa6f035-6d5b-4aa4-94fc-01385154ce58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz"
Sep 10 17:38:10.635008 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:10.634977    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa6f035-6d5b-4aa4-94fc-01385154ce58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz\" (UID: \"7fa6f035-6d5b-4aa4-94fc-01385154ce58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz"
Sep 10 17:38:10.635689 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:10.635663    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa6f035-6d5b-4aa4-94fc-01385154ce58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz\" (UID: \"7fa6f035-6d5b-4aa4-94fc-01385154ce58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz"
Sep 10 17:38:10.788967 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:10.788883    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz"
Sep 10 17:38:11.059184 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:11.059160    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz"]
Sep 10 17:38:11.061204 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:38:11.061179    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa6f035_6d5b_4aa4_94fc_01385154ce58.slice/crio-f93370a5ae6c36c57a21bacf91016d91fd835b97d81730d27abaa9b82d67e9dd WatchSource:0}: Error finding container f93370a5ae6c36c57a21bacf91016d91fd835b97d81730d27abaa9b82d67e9dd: Status 404 returned error can't find the container with id f93370a5ae6c36c57a21bacf91016d91fd835b97d81730d27abaa9b82d67e9dd
Sep 10 17:38:12.003351 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:12.003312    2552 generic.go:356] "Generic (PLEG): container finished" podID="7fa6f035-6d5b-4aa4-94fc-01385154ce58" containerID="309384291ddac0b971c8cd353866c5e3ab0fd38eb53a6a74b7aad180af508756" exitCode=0
Sep 10 17:38:12.003846 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:12.003391    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz" event={"ID":"7fa6f035-6d5b-4aa4-94fc-01385154ce58","Type":"ContainerDied","Data":"309384291ddac0b971c8cd353866c5e3ab0fd38eb53a6a74b7aad180af508756"}
Sep 10 17:38:12.003846 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:12.003446    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz" event={"ID":"7fa6f035-6d5b-4aa4-94fc-01385154ce58","Type":"ContainerStarted","Data":"f93370a5ae6c36c57a21bacf91016d91fd835b97d81730d27abaa9b82d67e9dd"}
Sep 10 17:38:13.147852 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:13.147829    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz"
Sep 10 17:38:13.156992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:13.156959    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa6f035-6d5b-4aa4-94fc-01385154ce58-maas-api-service-ca\") pod \"7fa6f035-6d5b-4aa4-94fc-01385154ce58\" (UID: \"7fa6f035-6d5b-4aa4-94fc-01385154ce58\") "
Sep 10 17:38:13.157269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:13.157248    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa6f035-6d5b-4aa4-94fc-01385154ce58-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7fa6f035-6d5b-4aa4-94fc-01385154ce58" (UID: "7fa6f035-6d5b-4aa4-94fc-01385154ce58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:38:13.157355 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:13.157337    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa6f035-6d5b-4aa4-94fc-01385154ce58-maas-api-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:14.015511 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:14.015476    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz"
Sep 10 17:38:14.015658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:14.015513    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz" event={"ID":"7fa6f035-6d5b-4aa4-94fc-01385154ce58","Type":"ContainerDied","Data":"f93370a5ae6c36c57a21bacf91016d91fd835b97d81730d27abaa9b82d67e9dd"}
Sep 10 17:38:14.015658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:14.015557    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f93370a5ae6c36c57a21bacf91016d91fd835b97d81730d27abaa9b82d67e9dd"
Sep 10 17:38:16.591197 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:16.591158    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz"]
Sep 10 17:38:16.599834 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:16.599800    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0b7bb86-9dcrz"]
Sep 10 17:38:16.804558 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:16.804517    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fa6f035-6d5b-4aa4-94fc-01385154ce58" path="/var/lib/kubelet/pods/7fa6f035-6d5b-4aa4-94fc-01385154ce58/volumes"
Sep 10 17:38:17.023265 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:17.023161    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:17.023265 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:17.023208    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:17.037954 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:17.037925    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:17.049478 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:17.049445    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:18.423368 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.423336    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"]
Sep 10 17:38:18.423754 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.423741    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa6f035-6d5b-4aa4-94fc-01385154ce58" containerName="revoke-keys"
Sep 10 17:38:18.423754 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.423753    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa6f035-6d5b-4aa4-94fc-01385154ce58" containerName="revoke-keys"
Sep 10 17:38:18.423845 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.423835    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa6f035-6d5b-4aa4-94fc-01385154ce58" containerName="revoke-keys"
Sep 10 17:38:18.427091 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.427068    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.429851 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.429822    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e67dcae3-openshift-default-dockercfg-fxf98\""
Sep 10 17:38:18.437828 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.437800    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"]
Sep 10 17:38:18.511373 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.511335    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-data\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.511565 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.511398    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-podinfo\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.511565 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.511431    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-workload-socket\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.511565 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.511532    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-token\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.511701 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.511566    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-workload-certs\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.511701 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.511667    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-credential-socket\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.511780 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.511759    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjpgh\" (UniqueName: \"kubernetes.io/projected/f1973f7c-dd1d-4794-9d9d-b74743bdce78-kube-api-access-pjpgh\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.511865 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.511844    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-envoy\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.511922 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.511895    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istiod-ca-cert\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.613230 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.613193    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-data\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.613453 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.613249    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-podinfo\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.613453 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.613271    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-workload-socket\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.613453 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.613302    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-token\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.613453 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.613330    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-workload-certs\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.613453 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.613379    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-credential-socket\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.613757 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.613480    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjpgh\" (UniqueName: \"kubernetes.io/projected/f1973f7c-dd1d-4794-9d9d-b74743bdce78-kube-api-access-pjpgh\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.613757 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.613596    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-envoy\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.613757 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.613625    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istiod-ca-cert\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.613757 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.613677    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-data\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.613973 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.613843    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-credential-socket\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.613973 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.613930    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-workload-socket\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.614111 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.614091    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-workload-certs\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.614418 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.614391    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istiod-ca-cert\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.615880 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.615851    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-envoy\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.616213 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.616189    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-podinfo\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.630228 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.630199    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-token\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.641136 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.641106    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjpgh\" (UniqueName: \"kubernetes.io/projected/f1973f7c-dd1d-4794-9d9d-b74743bdce78-kube-api-access-pjpgh\") pod \"e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") " pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:18.741559 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:18.741443    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:19.026369 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:19.026333    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"]
Sep 10 17:38:19.026598 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:38:19.026573    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1973f7c_dd1d_4794_9d9d_b74743bdce78.slice/crio-8f679663f41542593cf2152815ad7d45b389e9ffda6704d0361dc7c98cb2c67e WatchSource:0}: Error finding container 8f679663f41542593cf2152815ad7d45b389e9ffda6704d0361dc7c98cb2c67e: Status 404 returned error can't find the container with id 8f679663f41542593cf2152815ad7d45b389e9ffda6704d0361dc7c98cb2c67e
Sep 10 17:38:19.028834 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:19.028801    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:38:19.028910 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:19.028865    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:38:19.028947 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:19.028912    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:38:19.046748 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:19.046708    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c" event={"ID":"f1973f7c-dd1d-4794-9d9d-b74743bdce78","Type":"ContainerStarted","Data":"8f679663f41542593cf2152815ad7d45b389e9ffda6704d0361dc7c98cb2c67e"}
Sep 10 17:38:20.053624 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:20.053586    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c" event={"ID":"f1973f7c-dd1d-4794-9d9d-b74743bdce78","Type":"ContainerStarted","Data":"88c828d0adb481a62c33dbd8d49cca563794aa0f66d6127ac70d9935cc2884f5"}
Sep 10 17:38:20.075613 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:20.075559    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c" podStartSLOduration=2.075542796 podStartE2EDuration="2.075542796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:20.07489687 +0000 UTC m=+871.833423976" watchObservedRunningTime="2026-09-10 17:38:20.075542796 +0000 UTC m=+871.834069866"
Sep 10 17:38:20.742068 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:20.742031    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:20.747165 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:20.747139    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:21.059375 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:21.059293    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:21.060529 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:21.060484    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:21.819011 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:21.818972    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"]
Sep 10 17:38:21.822557 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:21.822539    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:21.827561 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:21.827534    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0d25fa\"/\"kube-root-ca.crt\""
Sep 10 17:38:21.827698 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:21.827560    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0d25fa\"/\"openshift-service-ca.crt\""
Sep 10 17:38:21.827698 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:21.827534    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0d25fa\"/\"test-model-0d25fa-kserve-self-signed-certs\""
Sep 10 17:38:21.827698 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:21.827573    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0d25fa\"/\"default-dockercfg-dn8nq\""
Sep 10 17:38:21.840760 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:21.840728    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"]
Sep 10 17:38:21.950295 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:21.950254    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-dshm\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:21.950295 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:21.950294    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6sq5\" (UniqueName: \"kubernetes.io/projected/3d772bb2-787c-439a-8e3c-c082df619445-kube-api-access-z6sq5\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:21.950552 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:21.950324    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-tmp-dir\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:21.950552 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:21.950462    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-home\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:21.950552 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:21.950537    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d772bb2-787c-439a-8e3c-c082df619445-tls-certs\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:21.950670 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:21.950581    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-model-cache\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:22.051927 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:22.051886    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-home\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:22.051927 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:22.051931    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d772bb2-787c-439a-8e3c-c082df619445-tls-certs\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:22.052133 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:22.051992    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-model-cache\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:22.052133 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:22.052036    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-dshm\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:22.052133 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:22.052056    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6sq5\" (UniqueName: \"kubernetes.io/projected/3d772bb2-787c-439a-8e3c-c082df619445-kube-api-access-z6sq5\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:22.052133 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:22.052086    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-tmp-dir\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:22.052559 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:22.052535    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-model-cache\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:22.052559 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:22.052557    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-tmp-dir\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:22.052701 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:22.052592    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-home\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:22.054259 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:22.054240    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-dshm\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:22.054425 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:22.054409    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d772bb2-787c-439a-8e3c-c082df619445-tls-certs\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:22.061669 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:22.061641    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6sq5\" (UniqueName: \"kubernetes.io/projected/3d772bb2-787c-439a-8e3c-c082df619445-kube-api-access-z6sq5\") pod \"test-model-0d25fa-kserve-545657d6df-vwzsh\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") " pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:22.132596 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:22.132557    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:22.410054 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:22.410019    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"]
Sep 10 17:38:22.412556 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:38:22.412525    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d772bb2_787c_439a_8e3c_c082df619445.slice/crio-e2950dc546244e17a4acdd0c3184824249050c1f5577394d33e59d7c6f1918c0 WatchSource:0}: Error finding container e2950dc546244e17a4acdd0c3184824249050c1f5577394d33e59d7c6f1918c0: Status 404 returned error can't find the container with id e2950dc546244e17a4acdd0c3184824249050c1f5577394d33e59d7c6f1918c0
Sep 10 17:38:23.071731 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:23.071689    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh" event={"ID":"3d772bb2-787c-439a-8e3c-c082df619445","Type":"ContainerStarted","Data":"75b194483746c138e46dd4e54efa6224fb00d12f24cb8004934a6f8421e95ee6"}
Sep 10 17:38:23.072220 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:23.071746    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh" event={"ID":"3d772bb2-787c-439a-8e3c-c082df619445","Type":"ContainerStarted","Data":"e2950dc546244e17a4acdd0c3184824249050c1f5577394d33e59d7c6f1918c0"}
Sep 10 17:38:23.097451 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:23.097395    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh" podStartSLOduration=2.097378473 podStartE2EDuration="2.097378473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:23.096888316 +0000 UTC m=+874.855415388" watchObservedRunningTime="2026-09-10 17:38:23.097378473 +0000 UTC m=+874.855905543"
Sep 10 17:38:25.376157 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:25.376121    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"]
Sep 10 17:38:27.386047 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:27.386007    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"]
Sep 10 17:38:27.506883 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:27.506835    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:38:32.132694 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:32.132662    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:32.133099 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:32.132924    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:32.169152 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:32.169119    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:32.986735 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:32.986695    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"]
Sep 10 17:38:33.101621 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:33.101584    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967"]
Sep 10 17:38:33.105484 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:33.105467    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967"
Sep 10 17:38:33.108339 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:33.108311    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:38:33.108440 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:33.108377    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 17:38:33.112923 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:33.112897    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967"]
Sep 10 17:38:33.136652 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:33.136618    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:38:33.279093 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:33.279011    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4091923-7cc0-4ea7-8c08-8b10acd39004-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-794622-g9967\" (UID: \"d4091923-7cc0-4ea7-8c08-8b10acd39004\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967"
Sep 10 17:38:33.380408 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:33.380358    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4091923-7cc0-4ea7-8c08-8b10acd39004-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-794622-g9967\" (UID: \"d4091923-7cc0-4ea7-8c08-8b10acd39004\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967"
Sep 10 17:38:33.381024 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:33.381003    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4091923-7cc0-4ea7-8c08-8b10acd39004-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-794622-g9967\" (UID: \"d4091923-7cc0-4ea7-8c08-8b10acd39004\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967"
Sep 10 17:38:33.418820 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:33.418786    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967"
Sep 10 17:38:33.712221 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:33.712197    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967"]
Sep 10 17:38:33.714182 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:38:33.714156    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4091923_7cc0_4ea7_8c08_8b10acd39004.slice/crio-2a279ba6409d5edd7a88b7ab6e8a4cbbb7d9e19704ee12a958450cc53bc650e3 WatchSource:0}: Error finding container 2a279ba6409d5edd7a88b7ab6e8a4cbbb7d9e19704ee12a958450cc53bc650e3: Status 404 returned error can't find the container with id 2a279ba6409d5edd7a88b7ab6e8a4cbbb7d9e19704ee12a958450cc53bc650e3
Sep 10 17:38:34.132245 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:34.132206    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967" event={"ID":"d4091923-7cc0-4ea7-8c08-8b10acd39004","Type":"ContainerStarted","Data":"90fc4655b41ab0242751793981e63d6385307db24f22118a53f270e68a34484b"}
Sep 10 17:38:34.132409 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:34.132271    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967" event={"ID":"d4091923-7cc0-4ea7-8c08-8b10acd39004","Type":"ContainerStarted","Data":"2a279ba6409d5edd7a88b7ab6e8a4cbbb7d9e19704ee12a958450cc53bc650e3"}
Sep 10 17:38:34.153516 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:34.149909    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967" podStartSLOduration=1.149872993 podStartE2EDuration="1.149872993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:34.148137109 +0000 UTC m=+885.906664180" watchObservedRunningTime="2026-09-10 17:38:34.149872993 +0000 UTC m=+885.908400064"
Sep 10 17:38:35.138842 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:35.138808    2552 generic.go:356] "Generic (PLEG): container finished" podID="d4091923-7cc0-4ea7-8c08-8b10acd39004" containerID="90fc4655b41ab0242751793981e63d6385307db24f22118a53f270e68a34484b" exitCode=0
Sep 10 17:38:35.139030 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:35.138844    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967" event={"ID":"d4091923-7cc0-4ea7-8c08-8b10acd39004","Type":"ContainerDied","Data":"90fc4655b41ab0242751793981e63d6385307db24f22118a53f270e68a34484b"}
Sep 10 17:38:36.288337 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:36.288306    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967"
Sep 10 17:38:36.414590 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:36.414480    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4091923-7cc0-4ea7-8c08-8b10acd39004-maas-api-service-ca\") pod \"d4091923-7cc0-4ea7-8c08-8b10acd39004\" (UID: \"d4091923-7cc0-4ea7-8c08-8b10acd39004\") "
Sep 10 17:38:36.414788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:36.414764    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4091923-7cc0-4ea7-8c08-8b10acd39004-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4091923-7cc0-4ea7-8c08-8b10acd39004" (UID: "d4091923-7cc0-4ea7-8c08-8b10acd39004"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:38:36.414880 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:36.414867    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4091923-7cc0-4ea7-8c08-8b10acd39004-maas-api-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:37.101699 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:37.101652    2552 patch_prober.go:28] interesting pod/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 17:38:37.101898 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:37.101730    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c" podUID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 17:38:37.153639 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:37.153597    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967"
Sep 10 17:38:37.153834 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:37.153596    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967" event={"ID":"d4091923-7cc0-4ea7-8c08-8b10acd39004","Type":"ContainerDied","Data":"2a279ba6409d5edd7a88b7ab6e8a4cbbb7d9e19704ee12a958450cc53bc650e3"}
Sep 10 17:38:37.153834 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:37.153719    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a279ba6409d5edd7a88b7ab6e8a4cbbb7d9e19704ee12a958450cc53bc650e3"
Sep 10 17:38:39.166217 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:39.166180    2552 generic.go:356] "Generic (PLEG): container finished" podID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" containerID="88c828d0adb481a62c33dbd8d49cca563794aa0f66d6127ac70d9935cc2884f5" exitCode=0
Sep 10 17:38:39.166217 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:39.166221    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c" event={"ID":"f1973f7c-dd1d-4794-9d9d-b74743bdce78","Type":"ContainerDied","Data":"88c828d0adb481a62c33dbd8d49cca563794aa0f66d6127ac70d9935cc2884f5"}
Sep 10 17:38:39.166831 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:39.166799    2552 scope.go:117] "RemoveContainer" containerID="88c828d0adb481a62c33dbd8d49cca563794aa0f66d6127ac70d9935cc2884f5"
Sep 10 17:38:39.167961 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:39.167921    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:38:39.168058 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:39.167993    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:38:39.168058 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:39.168025    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:38:40.176139 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.176106    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c" event={"ID":"f1973f7c-dd1d-4794-9d9d-b74743bdce78","Type":"ContainerStarted","Data":"b71501951fec23bcfb68502e6ec7ba08c0f9fc98bcb5f28ff14adb48e130fafa"}
Sep 10 17:38:40.396772 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.396714    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp" podUID="4e16b75c-cffd-4af2-af38-ccdf202dc9fa" containerName="main" containerID="cri-o://ebf027d0c41b4c3745994f0c81aec55c5024b3af3296071aa3d7f9bdd7042c70" gracePeriod=45
Sep 10 17:38:40.697316 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.697289    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:38:40.741878 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.741849    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:40.742109 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.742089    2552 patch_prober.go:28] interesting pod/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 10 17:38:40.742154 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.742140    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c" podUID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 10 17:38:40.770264 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.770171    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-tls-certs\") pod \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") "
Sep 10 17:38:40.770264 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.770223    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-tmp-dir\") pod \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") "
Sep 10 17:38:40.770264 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.770265    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-dshm\") pod \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") "
Sep 10 17:38:40.770477 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.770319    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-home\") pod \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") "
Sep 10 17:38:40.770477 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.770416    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g4r2\" (UniqueName: \"kubernetes.io/projected/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-kube-api-access-7g4r2\") pod \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") "
Sep 10 17:38:40.770477 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.770462    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-model-cache\") pod \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\" (UID: \"4e16b75c-cffd-4af2-af38-ccdf202dc9fa\") "
Sep 10 17:38:40.770658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.770627    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-home" (OuterVolumeSpecName: "home") pod "4e16b75c-cffd-4af2-af38-ccdf202dc9fa" (UID: "4e16b75c-cffd-4af2-af38-ccdf202dc9fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:40.770891 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.770852    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e16b75c-cffd-4af2-af38-ccdf202dc9fa" (UID: "4e16b75c-cffd-4af2-af38-ccdf202dc9fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:40.771041 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.770973    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-home\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:40.771041 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.770992    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-model-cache\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:40.771157 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.771070    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e16b75c-cffd-4af2-af38-ccdf202dc9fa" (UID: "4e16b75c-cffd-4af2-af38-ccdf202dc9fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:40.772716 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.772687    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-dshm" (OuterVolumeSpecName: "dshm") pod "4e16b75c-cffd-4af2-af38-ccdf202dc9fa" (UID: "4e16b75c-cffd-4af2-af38-ccdf202dc9fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:40.772825 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.772708    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-kube-api-access-7g4r2" (OuterVolumeSpecName: "kube-api-access-7g4r2") pod "4e16b75c-cffd-4af2-af38-ccdf202dc9fa" (UID: "4e16b75c-cffd-4af2-af38-ccdf202dc9fa"). InnerVolumeSpecName "kube-api-access-7g4r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:40.772864 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.772848    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e16b75c-cffd-4af2-af38-ccdf202dc9fa" (UID: "4e16b75c-cffd-4af2-af38-ccdf202dc9fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:38:40.871711 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.871668    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-tls-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:40.871711 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.871700    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-tmp-dir\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:40.871711 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.871709    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-dshm\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:40.871711 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:40.871717    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g4r2\" (UniqueName: \"kubernetes.io/projected/4e16b75c-cffd-4af2-af38-ccdf202dc9fa-kube-api-access-7g4r2\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:41.182069 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:41.182034    2552 generic.go:356] "Generic (PLEG): container finished" podID="4e16b75c-cffd-4af2-af38-ccdf202dc9fa" containerID="ebf027d0c41b4c3745994f0c81aec55c5024b3af3296071aa3d7f9bdd7042c70" exitCode=0
Sep 10 17:38:41.182520 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:41.182100    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"
Sep 10 17:38:41.182520 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:41.182114    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp" event={"ID":"4e16b75c-cffd-4af2-af38-ccdf202dc9fa","Type":"ContainerDied","Data":"ebf027d0c41b4c3745994f0c81aec55c5024b3af3296071aa3d7f9bdd7042c70"}
Sep 10 17:38:41.182520 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:41.182157    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp" event={"ID":"4e16b75c-cffd-4af2-af38-ccdf202dc9fa","Type":"ContainerDied","Data":"78771d83f0a458e9986f3398d4352dd81d565c1959f073c608bf0d2c3dc7889c"}
Sep 10 17:38:41.182520 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:41.182178    2552 scope.go:117] "RemoveContainer" containerID="ebf027d0c41b4c3745994f0c81aec55c5024b3af3296071aa3d7f9bdd7042c70"
Sep 10 17:38:41.195676 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:41.195639    2552 scope.go:117] "RemoveContainer" containerID="ebf027d0c41b4c3745994f0c81aec55c5024b3af3296071aa3d7f9bdd7042c70"
Sep 10 17:38:41.195974 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:38:41.195943    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf027d0c41b4c3745994f0c81aec55c5024b3af3296071aa3d7f9bdd7042c70\": container with ID starting with ebf027d0c41b4c3745994f0c81aec55c5024b3af3296071aa3d7f9bdd7042c70 not found: ID does not exist" containerID="ebf027d0c41b4c3745994f0c81aec55c5024b3af3296071aa3d7f9bdd7042c70"
Sep 10 17:38:41.196043 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:41.195983    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf027d0c41b4c3745994f0c81aec55c5024b3af3296071aa3d7f9bdd7042c70"} err="failed to get container status \"ebf027d0c41b4c3745994f0c81aec55c5024b3af3296071aa3d7f9bdd7042c70\": rpc error: code = NotFound desc = could not find container \"ebf027d0c41b4c3745994f0c81aec55c5024b3af3296071aa3d7f9bdd7042c70\": container with ID starting with ebf027d0c41b4c3745994f0c81aec55c5024b3af3296071aa3d7f9bdd7042c70 not found: ID does not exist"
Sep 10 17:38:41.201242 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:41.201209    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"]
Sep 10 17:38:41.204773 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:41.204748    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-21724959-kserve-776c8498db-bdgvp"]
Sep 10 17:38:41.742289 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:41.742256    2552 patch_prober.go:28] interesting pod/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 10 17:38:41.742467 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:41.742320    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c" podUID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 10 17:38:42.405980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.405920    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25" podUID="1f3b6924-e415-4afc-9298-31c8aaf5f600" containerName="main" containerID="cri-o://c0147932453ca562716fd6d95c50a975de37db160d96427ea97c89a3655f6936" gracePeriod=45
Sep 10 17:38:42.699901 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.699874    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:38:42.742271 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.742235    2552 patch_prober.go:28] interesting pod/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 10 17:38:42.742430 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.742300    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c" podUID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 10 17:38:42.788223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.788186    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skxxb\" (UniqueName: \"kubernetes.io/projected/1f3b6924-e415-4afc-9298-31c8aaf5f600-kube-api-access-skxxb\") pod \"1f3b6924-e415-4afc-9298-31c8aaf5f600\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") "
Sep 10 17:38:42.788223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.788233    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-tmp-dir\") pod \"1f3b6924-e415-4afc-9298-31c8aaf5f600\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") "
Sep 10 17:38:42.788435 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.788254    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-dshm\") pod \"1f3b6924-e415-4afc-9298-31c8aaf5f600\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") "
Sep 10 17:38:42.788435 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.788306    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-model-cache\") pod \"1f3b6924-e415-4afc-9298-31c8aaf5f600\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") "
Sep 10 17:38:42.788435 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.788411    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b6924-e415-4afc-9298-31c8aaf5f600-tls-certs\") pod \"1f3b6924-e415-4afc-9298-31c8aaf5f600\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") "
Sep 10 17:38:42.788435 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.788428    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-home\") pod \"1f3b6924-e415-4afc-9298-31c8aaf5f600\" (UID: \"1f3b6924-e415-4afc-9298-31c8aaf5f600\") "
Sep 10 17:38:42.788609 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.788460    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f3b6924-e415-4afc-9298-31c8aaf5f600" (UID: "1f3b6924-e415-4afc-9298-31c8aaf5f600"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:42.788672 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.788639    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f3b6924-e415-4afc-9298-31c8aaf5f600" (UID: "1f3b6924-e415-4afc-9298-31c8aaf5f600"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:42.788737 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.788719    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-tmp-dir\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:42.788789 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.788743    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-model-cache\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:42.788841 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.788820    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-home" (OuterVolumeSpecName: "home") pod "1f3b6924-e415-4afc-9298-31c8aaf5f600" (UID: "1f3b6924-e415-4afc-9298-31c8aaf5f600"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:42.790548 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.790478    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3b6924-e415-4afc-9298-31c8aaf5f600-kube-api-access-skxxb" (OuterVolumeSpecName: "kube-api-access-skxxb") pod "1f3b6924-e415-4afc-9298-31c8aaf5f600" (UID: "1f3b6924-e415-4afc-9298-31c8aaf5f600"). InnerVolumeSpecName "kube-api-access-skxxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:42.790698 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.790621    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-dshm" (OuterVolumeSpecName: "dshm") pod "1f3b6924-e415-4afc-9298-31c8aaf5f600" (UID: "1f3b6924-e415-4afc-9298-31c8aaf5f600"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:42.791101 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.791081    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3b6924-e415-4afc-9298-31c8aaf5f600-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f3b6924-e415-4afc-9298-31c8aaf5f600" (UID: "1f3b6924-e415-4afc-9298-31c8aaf5f600"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:38:42.805004 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.804978    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e16b75c-cffd-4af2-af38-ccdf202dc9fa" path="/var/lib/kubelet/pods/4e16b75c-cffd-4af2-af38-ccdf202dc9fa/volumes"
Sep 10 17:38:42.890247 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.890200    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b6924-e415-4afc-9298-31c8aaf5f600-tls-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:42.890548 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.890526    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-home\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:42.890657 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.890648    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skxxb\" (UniqueName: \"kubernetes.io/projected/1f3b6924-e415-4afc-9298-31c8aaf5f600-kube-api-access-skxxb\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:42.890715 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:42.890707    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3b6924-e415-4afc-9298-31c8aaf5f600-dshm\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:43.194896 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:43.194862    2552 generic.go:356] "Generic (PLEG): container finished" podID="1f3b6924-e415-4afc-9298-31c8aaf5f600" containerID="c0147932453ca562716fd6d95c50a975de37db160d96427ea97c89a3655f6936" exitCode=0
Sep 10 17:38:43.195096 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:43.194910    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"
Sep 10 17:38:43.195096 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:43.194942    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25" event={"ID":"1f3b6924-e415-4afc-9298-31c8aaf5f600","Type":"ContainerDied","Data":"c0147932453ca562716fd6d95c50a975de37db160d96427ea97c89a3655f6936"}
Sep 10 17:38:43.195096 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:43.194979    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25" event={"ID":"1f3b6924-e415-4afc-9298-31c8aaf5f600","Type":"ContainerDied","Data":"87a463d420424d0dd34d09a8e9c62292faec78b29e078a993bb799f1bcd74f1d"}
Sep 10 17:38:43.195096 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:43.194996    2552 scope.go:117] "RemoveContainer" containerID="c0147932453ca562716fd6d95c50a975de37db160d96427ea97c89a3655f6936"
Sep 10 17:38:43.203559 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:43.203542    2552 scope.go:117] "RemoveContainer" containerID="c0147932453ca562716fd6d95c50a975de37db160d96427ea97c89a3655f6936"
Sep 10 17:38:43.203813 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:38:43.203795    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0147932453ca562716fd6d95c50a975de37db160d96427ea97c89a3655f6936\": container with ID starting with c0147932453ca562716fd6d95c50a975de37db160d96427ea97c89a3655f6936 not found: ID does not exist" containerID="c0147932453ca562716fd6d95c50a975de37db160d96427ea97c89a3655f6936"
Sep 10 17:38:43.203866 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:43.203832    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0147932453ca562716fd6d95c50a975de37db160d96427ea97c89a3655f6936"} err="failed to get container status \"c0147932453ca562716fd6d95c50a975de37db160d96427ea97c89a3655f6936\": rpc error: code = NotFound desc = could not find container \"c0147932453ca562716fd6d95c50a975de37db160d96427ea97c89a3655f6936\": container with ID starting with c0147932453ca562716fd6d95c50a975de37db160d96427ea97c89a3655f6936 not found: ID does not exist"
Sep 10 17:38:43.226787 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:43.226762    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"]
Sep 10 17:38:43.235411 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:43.235386    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-21724959-kserve-69868dc9b-p7d25"]
Sep 10 17:38:43.742161 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:43.742122    2552 patch_prober.go:28] interesting pod/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 10 17:38:43.742659 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:43.742190    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c" podUID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 10 17:38:44.741859 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:44.741821    2552 patch_prober.go:28] interesting pod/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 10 17:38:44.742012 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:44.741887    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c" podUID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 10 17:38:44.803840 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:44.803804    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f3b6924-e415-4afc-9298-31c8aaf5f600" path="/var/lib/kubelet/pods/1f3b6924-e415-4afc-9298-31c8aaf5f600/volumes"
Sep 10 17:38:44.810985 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:44.810955    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967"]
Sep 10 17:38:44.814315 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:44.814292    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-794622-g9967"]
Sep 10 17:38:45.745751 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:45.745719    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:45.746124 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:45.746099    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:45.746764 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:45.746743    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:46.242660 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.242622    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd"]
Sep 10 17:38:46.243039 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.243026    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4091923-7cc0-4ea7-8c08-8b10acd39004" containerName="revoke-keys"
Sep 10 17:38:46.243039 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.243040    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4091923-7cc0-4ea7-8c08-8b10acd39004" containerName="revoke-keys"
Sep 10 17:38:46.243119 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.243053    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3b6924-e415-4afc-9298-31c8aaf5f600" containerName="main"
Sep 10 17:38:46.243119 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.243060    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3b6924-e415-4afc-9298-31c8aaf5f600" containerName="main"
Sep 10 17:38:46.243119 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.243073    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e16b75c-cffd-4af2-af38-ccdf202dc9fa" containerName="main"
Sep 10 17:38:46.243119 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.243079    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e16b75c-cffd-4af2-af38-ccdf202dc9fa" containerName="main"
Sep 10 17:38:46.243242 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.243143    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4091923-7cc0-4ea7-8c08-8b10acd39004" containerName="revoke-keys"
Sep 10 17:38:46.243242 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.243151    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e16b75c-cffd-4af2-af38-ccdf202dc9fa" containerName="main"
Sep 10 17:38:46.243242 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.243158    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f3b6924-e415-4afc-9298-31c8aaf5f600" containerName="main"
Sep 10 17:38:46.245242 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.245225    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd"
Sep 10 17:38:46.247750 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.247724    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 17:38:46.247750 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.247733    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:38:46.253649 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.253626    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd"]
Sep 10 17:38:46.324242 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.324210    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a1190b-cba9-47f2-be22-d5ccc5ee1057-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd\" (UID: \"47a1190b-cba9-47f2-be22-d5ccc5ee1057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd"
Sep 10 17:38:46.425616 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.425571    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a1190b-cba9-47f2-be22-d5ccc5ee1057-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd\" (UID: \"47a1190b-cba9-47f2-be22-d5ccc5ee1057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd"
Sep 10 17:38:46.426210 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.426189    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a1190b-cba9-47f2-be22-d5ccc5ee1057-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd\" (UID: \"47a1190b-cba9-47f2-be22-d5ccc5ee1057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd"
Sep 10 17:38:46.556788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.556691    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd"
Sep 10 17:38:46.803427 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.803389    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4091923-7cc0-4ea7-8c08-8b10acd39004" path="/var/lib/kubelet/pods/d4091923-7cc0-4ea7-8c08-8b10acd39004/volumes"
Sep 10 17:38:46.832394 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:46.832364    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd"]
Sep 10 17:38:46.834426 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:38:46.834393    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a1190b_cba9_47f2_be22_d5ccc5ee1057.slice/crio-9ca7d3424354042501eea1f89657b11716189b6ccdf0308197799046853c7e25 WatchSource:0}: Error finding container 9ca7d3424354042501eea1f89657b11716189b6ccdf0308197799046853c7e25: Status 404 returned error can't find the container with id 9ca7d3424354042501eea1f89657b11716189b6ccdf0308197799046853c7e25
Sep 10 17:38:47.219343 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:47.219305    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd" event={"ID":"47a1190b-cba9-47f2-be22-d5ccc5ee1057","Type":"ContainerStarted","Data":"bb2d802b2982674d91ffd4f7d653885410a90ca41379d5d76cadca6ad32e9277"}
Sep 10 17:38:47.219540 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:47.219362    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd" event={"ID":"47a1190b-cba9-47f2-be22-d5ccc5ee1057","Type":"ContainerStarted","Data":"9ca7d3424354042501eea1f89657b11716189b6ccdf0308197799046853c7e25"}
Sep 10 17:38:47.240770 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:47.240722    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd" podStartSLOduration=1.24070773 podStartE2EDuration="1.24070773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:47.23893859 +0000 UTC m=+898.997465663" watchObservedRunningTime="2026-09-10 17:38:47.24070773 +0000 UTC m=+898.999234800"
Sep 10 17:38:47.399099 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:47.399058    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:38:48.007413 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.007356    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj" podUID="1396e850-ed20-41d3-ac05-b9c7c3275e8d" containerName="main" containerID="cri-o://48687698977e3a013dbedc86782cb992949d3b6df435e7acc4e1663b008f4f2d" gracePeriod=45
Sep 10 17:38:48.226508 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.226461    2552 generic.go:356] "Generic (PLEG): container finished" podID="47a1190b-cba9-47f2-be22-d5ccc5ee1057" containerID="bb2d802b2982674d91ffd4f7d653885410a90ca41379d5d76cadca6ad32e9277" exitCode=0
Sep 10 17:38:48.226676 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.226525    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd" event={"ID":"47a1190b-cba9-47f2-be22-d5ccc5ee1057","Type":"ContainerDied","Data":"bb2d802b2982674d91ffd4f7d653885410a90ca41379d5d76cadca6ad32e9277"}
Sep 10 17:38:48.228363 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.228339    2552 generic.go:356] "Generic (PLEG): container finished" podID="1396e850-ed20-41d3-ac05-b9c7c3275e8d" containerID="48687698977e3a013dbedc86782cb992949d3b6df435e7acc4e1663b008f4f2d" exitCode=0
Sep 10 17:38:48.228534 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.228407    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj" event={"ID":"1396e850-ed20-41d3-ac05-b9c7c3275e8d","Type":"ContainerDied","Data":"48687698977e3a013dbedc86782cb992949d3b6df435e7acc4e1663b008f4f2d"}
Sep 10 17:38:48.274671 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.274602    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:48.344904 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.344866    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1396e850-ed20-41d3-ac05-b9c7c3275e8d-tls-certs\") pod \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") "
Sep 10 17:38:48.345107 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.344949    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp2vp\" (UniqueName: \"kubernetes.io/projected/1396e850-ed20-41d3-ac05-b9c7c3275e8d-kube-api-access-hp2vp\") pod \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") "
Sep 10 17:38:48.345107 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.344967    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-tmp-dir\") pod \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") "
Sep 10 17:38:48.345107 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.344994    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-home\") pod \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") "
Sep 10 17:38:48.345107 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.345009    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-dshm\") pod \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") "
Sep 10 17:38:48.345331 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.345142    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-model-cache\") pod \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\" (UID: \"1396e850-ed20-41d3-ac05-b9c7c3275e8d\") "
Sep 10 17:38:48.345331 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.345291    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-home" (OuterVolumeSpecName: "home") pod "1396e850-ed20-41d3-ac05-b9c7c3275e8d" (UID: "1396e850-ed20-41d3-ac05-b9c7c3275e8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:48.345331 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.345298    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1396e850-ed20-41d3-ac05-b9c7c3275e8d" (UID: "1396e850-ed20-41d3-ac05-b9c7c3275e8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:48.345526 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.345393    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "1396e850-ed20-41d3-ac05-b9c7c3275e8d" (UID: "1396e850-ed20-41d3-ac05-b9c7c3275e8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:48.345579 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.345524    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-tmp-dir\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:48.345579 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.345537    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-home\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:48.345579 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.345546    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-model-cache\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:48.347203 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.347183    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1396e850-ed20-41d3-ac05-b9c7c3275e8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1396e850-ed20-41d3-ac05-b9c7c3275e8d" (UID: "1396e850-ed20-41d3-ac05-b9c7c3275e8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:38:48.347571 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.347551    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-dshm" (OuterVolumeSpecName: "dshm") pod "1396e850-ed20-41d3-ac05-b9c7c3275e8d" (UID: "1396e850-ed20-41d3-ac05-b9c7c3275e8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:48.347649 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.347602    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1396e850-ed20-41d3-ac05-b9c7c3275e8d-kube-api-access-hp2vp" (OuterVolumeSpecName: "kube-api-access-hp2vp") pod "1396e850-ed20-41d3-ac05-b9c7c3275e8d" (UID: "1396e850-ed20-41d3-ac05-b9c7c3275e8d"). InnerVolumeSpecName "kube-api-access-hp2vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:48.446192 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.446144    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1396e850-ed20-41d3-ac05-b9c7c3275e8d-tls-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:48.446192 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.446178    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp2vp\" (UniqueName: \"kubernetes.io/projected/1396e850-ed20-41d3-ac05-b9c7c3275e8d-kube-api-access-hp2vp\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:48.446192 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.446188    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1396e850-ed20-41d3-ac05-b9c7c3275e8d-dshm\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:48.697835 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.697804    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:38:48.698784 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:48.698762    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:38:49.235412 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:49.235382    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"
Sep 10 17:38:49.235625 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:49.235409    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj" event={"ID":"1396e850-ed20-41d3-ac05-b9c7c3275e8d","Type":"ContainerDied","Data":"b5e7545c32cf5bba00c45070c16ecba4d51903c189e0e3e70f9624d14bbae18b"}
Sep 10 17:38:49.235625 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:49.235461    2552 scope.go:117] "RemoveContainer" containerID="48687698977e3a013dbedc86782cb992949d3b6df435e7acc4e1663b008f4f2d"
Sep 10 17:38:49.253016 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:49.252984    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"]
Sep 10 17:38:49.261579 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:49.261549    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-794622/autores-model-794622-kserve-8b589975f-l9hvj"]
Sep 10 17:38:49.370729 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:49.370702    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd"
Sep 10 17:38:49.456549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:49.456478    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a1190b-cba9-47f2-be22-d5ccc5ee1057-maas-api-service-ca\") pod \"47a1190b-cba9-47f2-be22-d5ccc5ee1057\" (UID: \"47a1190b-cba9-47f2-be22-d5ccc5ee1057\") "
Sep 10 17:38:49.456921 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:49.456804    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a1190b-cba9-47f2-be22-d5ccc5ee1057-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47a1190b-cba9-47f2-be22-d5ccc5ee1057" (UID: "47a1190b-cba9-47f2-be22-d5ccc5ee1057"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:38:49.558261 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:49.558163    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a1190b-cba9-47f2-be22-d5ccc5ee1057-maas-api-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:50.241768 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:50.241735    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd"
Sep 10 17:38:50.241926 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:50.241746    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd" event={"ID":"47a1190b-cba9-47f2-be22-d5ccc5ee1057","Type":"ContainerDied","Data":"9ca7d3424354042501eea1f89657b11716189b6ccdf0308197799046853c7e25"}
Sep 10 17:38:50.241926 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:50.241855    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ca7d3424354042501eea1f89657b11716189b6ccdf0308197799046853c7e25"
Sep 10 17:38:50.803534 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:50.803483    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1396e850-ed20-41d3-ac05-b9c7c3275e8d" path="/var/lib/kubelet/pods/1396e850-ed20-41d3-ac05-b9c7c3275e8d/volumes"
Sep 10 17:38:52.395648 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.395615    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"]
Sep 10 17:38:52.396083 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.396069    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1396e850-ed20-41d3-ac05-b9c7c3275e8d" containerName="main"
Sep 10 17:38:52.396083 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.396084    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1396e850-ed20-41d3-ac05-b9c7c3275e8d" containerName="main"
Sep 10 17:38:52.396175 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.396098    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47a1190b-cba9-47f2-be22-d5ccc5ee1057" containerName="revoke-keys"
Sep 10 17:38:52.396175 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.396104    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="47a1190b-cba9-47f2-be22-d5ccc5ee1057" containerName="revoke-keys"
Sep 10 17:38:52.396175 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.396165    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="47a1190b-cba9-47f2-be22-d5ccc5ee1057" containerName="revoke-keys"
Sep 10 17:38:52.396175 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.396175    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1396e850-ed20-41d3-ac05-b9c7c3275e8d" containerName="main"
Sep 10 17:38:52.398786 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.398764    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.402145 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.402120    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-14b3e7-openshift-default-dockercfg-xqvjt\""
Sep 10 17:38:52.417626 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.417594    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"]
Sep 10 17:38:52.486148 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.486111    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-token\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.486360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.486193    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5h2l\" (UniqueName: \"kubernetes.io/projected/289be72d-8c4c-4188-b86e-ce5dfd88ae92-kube-api-access-m5h2l\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.486360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.486269    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-podinfo\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.486360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.486301    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-workload-certs\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.486360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.486328    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-workload-socket\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.486557 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.486420    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-credential-socket\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.486557 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.486470    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-envoy\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.486557 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.486524    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-data\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.486557 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.486548    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istiod-ca-cert\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.587611 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.587571    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-envoy\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.587611 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.587616    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-data\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.587820 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.587631    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istiod-ca-cert\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.587855 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.587823    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-token\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.587946 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.587929    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5h2l\" (UniqueName: \"kubernetes.io/projected/289be72d-8c4c-4188-b86e-ce5dfd88ae92-kube-api-access-m5h2l\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.588013 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.587994    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-podinfo\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.588225 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.588199    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-workload-certs\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.588324 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.588248    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-workload-socket\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.588324 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.588073    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-data\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.588324 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.588286    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-credential-socket\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.588511 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.588334    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istiod-ca-cert\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.588511 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.588473    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-workload-certs\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.588614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.588570    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-credential-socket\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.588693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.588663    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-workload-socket\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.590073 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.590050    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-envoy\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.590383 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.590365    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-podinfo\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.596355 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.596326    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5h2l\" (UniqueName: \"kubernetes.io/projected/289be72d-8c4c-4188-b86e-ce5dfd88ae92-kube-api-access-m5h2l\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.596462 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.596444    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-token\") pod \"e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") " pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.710148 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.710052    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:52.994140 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.994113    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"]
Sep 10 17:38:52.995071 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:38:52.995047    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289be72d_8c4c_4188_b86e_ce5dfd88ae92.slice/crio-9dc098b55fa23b99a0064efeb689e909689d1cda9222a6d6d739beb171c678ae WatchSource:0}: Error finding container 9dc098b55fa23b99a0064efeb689e909689d1cda9222a6d6d739beb171c678ae: Status 404 returned error can't find the container with id 9dc098b55fa23b99a0064efeb689e909689d1cda9222a6d6d739beb171c678ae
Sep 10 17:38:52.997724 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.997685    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:38:52.997841 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.997773    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:38:52.997841 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:52.997820    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:38:53.262099 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:53.262003    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl" event={"ID":"289be72d-8c4c-4188-b86e-ce5dfd88ae92","Type":"ContainerStarted","Data":"f797d22925bb14df253ea6ca598bb8c2c1b1ecf02ea21d92801a68b4b240b1f2"}
Sep 10 17:38:53.262099 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:53.262066    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl" event={"ID":"289be72d-8c4c-4188-b86e-ce5dfd88ae92","Type":"ContainerStarted","Data":"9dc098b55fa23b99a0064efeb689e909689d1cda9222a6d6d739beb171c678ae"}
Sep 10 17:38:53.286048 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:53.285984    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl" podStartSLOduration=1.285955883 podStartE2EDuration="1.285955883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:53.282563319 +0000 UTC m=+905.041090393" watchObservedRunningTime="2026-09-10 17:38:53.285955883 +0000 UTC m=+905.044482955"
Sep 10 17:38:53.497659 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:53.497544    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd"]
Sep 10 17:38:53.499450 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:53.499426    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e67dcae3-dhrnd"]
Sep 10 17:38:53.634672 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:53.634631    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"]
Sep 10 17:38:53.634945 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:53.634889    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c" podUID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" containerName="istio-proxy" containerID="cri-o://b71501951fec23bcfb68502e6ec7ba08c0f9fc98bcb5f28ff14adb48e130fafa" gracePeriod=30
Sep 10 17:38:53.710215 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:53.710176    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:54.715260 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:54.715226    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:54.803741 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:54.803701    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47a1190b-cba9-47f2-be22-d5ccc5ee1057" path="/var/lib/kubelet/pods/47a1190b-cba9-47f2-be22-d5ccc5ee1057/volumes"
Sep 10 17:38:55.273113 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:55.273083    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:55.274016 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:55.273996    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:38:58.890184 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.890158    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:58.949671 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.949587    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjpgh\" (UniqueName: \"kubernetes.io/projected/f1973f7c-dd1d-4794-9d9d-b74743bdce78-kube-api-access-pjpgh\") pod \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") "
Sep 10 17:38:58.949671 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.949635    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-credential-socket\") pod \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") "
Sep 10 17:38:58.949671 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.949664    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-data\") pod \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") "
Sep 10 17:38:58.949896 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.949765    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istiod-ca-cert\") pod \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") "
Sep 10 17:38:58.949896 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.949853    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-podinfo\") pod \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") "
Sep 10 17:38:58.949896 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.949882    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-workload-socket\") pod \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") "
Sep 10 17:38:58.949896 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.949886    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-data" (OuterVolumeSpecName: "istio-data") pod "f1973f7c-dd1d-4794-9d9d-b74743bdce78" (UID: "f1973f7c-dd1d-4794-9d9d-b74743bdce78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:58.950065 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.949938    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-envoy\") pod \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") "
Sep 10 17:38:58.950065 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.949970    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-token\") pod \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") "
Sep 10 17:38:58.950065 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.949976    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f1973f7c-dd1d-4794-9d9d-b74743bdce78" (UID: "f1973f7c-dd1d-4794-9d9d-b74743bdce78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:58.950065 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.950000    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-workload-certs\") pod \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\" (UID: \"f1973f7c-dd1d-4794-9d9d-b74743bdce78\") "
Sep 10 17:38:58.950263 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.950084    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f1973f7c-dd1d-4794-9d9d-b74743bdce78" (UID: "f1973f7c-dd1d-4794-9d9d-b74743bdce78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:38:58.950263 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.950241    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f1973f7c-dd1d-4794-9d9d-b74743bdce78" (UID: "f1973f7c-dd1d-4794-9d9d-b74743bdce78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:58.950370 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.950324    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f1973f7c-dd1d-4794-9d9d-b74743bdce78" (UID: "f1973f7c-dd1d-4794-9d9d-b74743bdce78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:58.950517 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.950477    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-workload-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:58.950667 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.950528    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-credential-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:58.950667 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.950548    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-data\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:58.950667 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.950562    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istiod-ca-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:58.950667 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.950574    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-workload-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:58.951991 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.951963    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f1973f7c-dd1d-4794-9d9d-b74743bdce78" (UID: "f1973f7c-dd1d-4794-9d9d-b74743bdce78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:38:58.952114 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.951998    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1973f7c-dd1d-4794-9d9d-b74743bdce78-kube-api-access-pjpgh" (OuterVolumeSpecName: "kube-api-access-pjpgh") pod "f1973f7c-dd1d-4794-9d9d-b74743bdce78" (UID: "f1973f7c-dd1d-4794-9d9d-b74743bdce78"). InnerVolumeSpecName "kube-api-access-pjpgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:58.952114 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.952019    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-token" (OuterVolumeSpecName: "istio-token") pod "f1973f7c-dd1d-4794-9d9d-b74743bdce78" (UID: "f1973f7c-dd1d-4794-9d9d-b74743bdce78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:58.952269 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:58.952252    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f1973f7c-dd1d-4794-9d9d-b74743bdce78" (UID: "f1973f7c-dd1d-4794-9d9d-b74743bdce78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:59.051111 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.051067    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjpgh\" (UniqueName: \"kubernetes.io/projected/f1973f7c-dd1d-4794-9d9d-b74743bdce78-kube-api-access-pjpgh\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:59.051111 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.051104    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-podinfo\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:59.051111 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.051113    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-envoy\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:59.051111 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.051122    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1973f7c-dd1d-4794-9d9d-b74743bdce78-istio-token\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:38:59.297228 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.297136    2552 generic.go:356] "Generic (PLEG): container finished" podID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" containerID="b71501951fec23bcfb68502e6ec7ba08c0f9fc98bcb5f28ff14adb48e130fafa" exitCode=0
Sep 10 17:38:59.297228 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.297179    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"
Sep 10 17:38:59.297228 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.297196    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c" event={"ID":"f1973f7c-dd1d-4794-9d9d-b74743bdce78","Type":"ContainerDied","Data":"b71501951fec23bcfb68502e6ec7ba08c0f9fc98bcb5f28ff14adb48e130fafa"}
Sep 10 17:38:59.297471 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.297244    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c" event={"ID":"f1973f7c-dd1d-4794-9d9d-b74743bdce78","Type":"ContainerDied","Data":"8f679663f41542593cf2152815ad7d45b389e9ffda6704d0361dc7c98cb2c67e"}
Sep 10 17:38:59.297471 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.297267    2552 scope.go:117] "RemoveContainer" containerID="b71501951fec23bcfb68502e6ec7ba08c0f9fc98bcb5f28ff14adb48e130fafa"
Sep 10 17:38:59.307629 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.307608    2552 scope.go:117] "RemoveContainer" containerID="88c828d0adb481a62c33dbd8d49cca563794aa0f66d6127ac70d9935cc2884f5"
Sep 10 17:38:59.316120 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.316098    2552 scope.go:117] "RemoveContainer" containerID="b71501951fec23bcfb68502e6ec7ba08c0f9fc98bcb5f28ff14adb48e130fafa"
Sep 10 17:38:59.316400 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:38:59.316380    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71501951fec23bcfb68502e6ec7ba08c0f9fc98bcb5f28ff14adb48e130fafa\": container with ID starting with b71501951fec23bcfb68502e6ec7ba08c0f9fc98bcb5f28ff14adb48e130fafa not found: ID does not exist" containerID="b71501951fec23bcfb68502e6ec7ba08c0f9fc98bcb5f28ff14adb48e130fafa"
Sep 10 17:38:59.316469 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.316408    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71501951fec23bcfb68502e6ec7ba08c0f9fc98bcb5f28ff14adb48e130fafa"} err="failed to get container status \"b71501951fec23bcfb68502e6ec7ba08c0f9fc98bcb5f28ff14adb48e130fafa\": rpc error: code = NotFound desc = could not find container \"b71501951fec23bcfb68502e6ec7ba08c0f9fc98bcb5f28ff14adb48e130fafa\": container with ID starting with b71501951fec23bcfb68502e6ec7ba08c0f9fc98bcb5f28ff14adb48e130fafa not found: ID does not exist"
Sep 10 17:38:59.316469 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.316426    2552 scope.go:117] "RemoveContainer" containerID="88c828d0adb481a62c33dbd8d49cca563794aa0f66d6127ac70d9935cc2884f5"
Sep 10 17:38:59.316709 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:38:59.316688    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c828d0adb481a62c33dbd8d49cca563794aa0f66d6127ac70d9935cc2884f5\": container with ID starting with 88c828d0adb481a62c33dbd8d49cca563794aa0f66d6127ac70d9935cc2884f5 not found: ID does not exist" containerID="88c828d0adb481a62c33dbd8d49cca563794aa0f66d6127ac70d9935cc2884f5"
Sep 10 17:38:59.316747 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.316716    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c828d0adb481a62c33dbd8d49cca563794aa0f66d6127ac70d9935cc2884f5"} err="failed to get container status \"88c828d0adb481a62c33dbd8d49cca563794aa0f66d6127ac70d9935cc2884f5\": rpc error: code = NotFound desc = could not find container \"88c828d0adb481a62c33dbd8d49cca563794aa0f66d6127ac70d9935cc2884f5\": container with ID starting with 88c828d0adb481a62c33dbd8d49cca563794aa0f66d6127ac70d9935cc2884f5 not found: ID does not exist"
Sep 10 17:38:59.321897 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.321870    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"]
Sep 10 17:38:59.328174 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:38:59.328149    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e67dcae3-openshift-default-6f77ddf8b4-9d77c"]
Sep 10 17:39:00.803795 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:00.803758    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" path="/var/lib/kubelet/pods/f1973f7c-dd1d-4794-9d9d-b74743bdce78/volumes"
Sep 10 17:39:05.502848 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:05.502810    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:39:14.504978 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.504939    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"]
Sep 10 17:39:14.505588 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.505571    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" containerName="istio-proxy"
Sep 10 17:39:14.505657 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.505592    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" containerName="istio-proxy"
Sep 10 17:39:14.505657 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.505615    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" containerName="istio-proxy"
Sep 10 17:39:14.505657 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.505624    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" containerName="istio-proxy"
Sep 10 17:39:14.505759 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.505716    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" containerName="istio-proxy"
Sep 10 17:39:14.505759 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.505728    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1973f7c-dd1d-4794-9d9d-b74743bdce78" containerName="istio-proxy"
Sep 10 17:39:14.509068 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.509043    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.513151 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.513122    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2f1093\"/\"kube-root-ca.crt\""
Sep 10 17:39:14.513374 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.513127    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2f1093\"/\"openshift-service-ca.crt\""
Sep 10 17:39:14.513374 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.513132    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2f1093\"/\"default-dockercfg-76m4b\""
Sep 10 17:39:14.513374 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.513134    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2f1093\"/\"test-model-2f1093-kserve-self-signed-certs\""
Sep 10 17:39:14.528592 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.523040    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"]
Sep 10 17:39:14.612940 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.612899    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45zxm\" (UniqueName: \"kubernetes.io/projected/67945228-9082-4a56-921c-516aec05cfcf-kube-api-access-45zxm\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.613151 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.612958    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-model-cache\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.613151 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.613056    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-home\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.613151 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.613112    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-tmp-dir\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.613333 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.613181    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67945228-9082-4a56-921c-516aec05cfcf-tls-certs\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.613333 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.613236    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-dshm\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.713873 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.713837    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-home\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.714049 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.713891    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-tmp-dir\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.714049 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.713946    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67945228-9082-4a56-921c-516aec05cfcf-tls-certs\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.714049 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.713986    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-dshm\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.714049 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.714039    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45zxm\" (UniqueName: \"kubernetes.io/projected/67945228-9082-4a56-921c-516aec05cfcf-kube-api-access-45zxm\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.714273 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.714092    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-model-cache\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.714366 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.714338    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-tmp-dir\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.714586 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.714392    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-home\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.714586 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.714409    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-model-cache\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.716342 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.716320    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-dshm\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.716536 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.716511    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67945228-9082-4a56-921c-516aec05cfcf-tls-certs\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.722099 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.722076    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45zxm\" (UniqueName: \"kubernetes.io/projected/67945228-9082-4a56-921c-516aec05cfcf-kube-api-access-45zxm\") pod \"test-model-2f1093-kserve-6b8969dd75-9spnm\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") " pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:14.821715 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:14.821620    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:15.106553 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:15.106331    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"]
Sep 10 17:39:15.108766 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:39:15.108740    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67945228_9082_4a56_921c_516aec05cfcf.slice/crio-c932ec3ddb16019f341d962bdcba09c2d04682acc3d1fc60ea34e83652d14a0e WatchSource:0}: Error finding container c932ec3ddb16019f341d962bdcba09c2d04682acc3d1fc60ea34e83652d14a0e: Status 404 returned error can't find the container with id c932ec3ddb16019f341d962bdcba09c2d04682acc3d1fc60ea34e83652d14a0e
Sep 10 17:39:15.392023 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:15.391938    2552 generic.go:356] "Generic (PLEG): container finished" podID="289be72d-8c4c-4188-b86e-ce5dfd88ae92" containerID="f797d22925bb14df253ea6ca598bb8c2c1b1ecf02ea21d92801a68b4b240b1f2" exitCode=0
Sep 10 17:39:15.392187 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:15.392013    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl" event={"ID":"289be72d-8c4c-4188-b86e-ce5dfd88ae92","Type":"ContainerDied","Data":"f797d22925bb14df253ea6ca598bb8c2c1b1ecf02ea21d92801a68b4b240b1f2"}
Sep 10 17:39:15.392535 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:15.392489    2552 scope.go:117] "RemoveContainer" containerID="f797d22925bb14df253ea6ca598bb8c2c1b1ecf02ea21d92801a68b4b240b1f2"
Sep 10 17:39:15.393703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:15.393680    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm" event={"ID":"67945228-9082-4a56-921c-516aec05cfcf","Type":"ContainerStarted","Data":"ed0644c1c63dc7df6f0ecbc7308d3179a50baf35580713c62819c267e4e40f76"}
Sep 10 17:39:15.393830 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:15.393717    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm" event={"ID":"67945228-9082-4a56-921c-516aec05cfcf","Type":"ContainerStarted","Data":"c932ec3ddb16019f341d962bdcba09c2d04682acc3d1fc60ea34e83652d14a0e"}
Sep 10 17:39:15.393830 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:15.393677    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:39:15.393830 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:15.393799    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:39:15.393997 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:15.393854    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:39:15.438048 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:15.437988    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm" podStartSLOduration=1.4379674709999999 podStartE2EDuration="1.437967471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:39:15.435385466 +0000 UTC m=+927.193912538" watchObservedRunningTime="2026-09-10 17:39:15.437967471 +0000 UTC m=+927.196494542"
Sep 10 17:39:16.401016 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:16.400984    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl" event={"ID":"289be72d-8c4c-4188-b86e-ce5dfd88ae92","Type":"ContainerStarted","Data":"c4f5d96670783dde3090a413660374ed62e8aefb6d2111c071bbae5c36bdee7a"}
Sep 10 17:39:16.710833 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:16.710732    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:39:16.711032 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:16.711009    2552 patch_prober.go:28] interesting pod/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 10 17:39:16.711083 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:16.711058    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl" podUID="289be72d-8c4c-4188-b86e-ce5dfd88ae92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 10 17:39:17.710952 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:17.710915    2552 patch_prober.go:28] interesting pod/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 10 17:39:17.711424 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:17.710982    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl" podUID="289be72d-8c4c-4188-b86e-ce5dfd88ae92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 10 17:39:18.711489 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:18.711450    2552 patch_prober.go:28] interesting pod/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 10 17:39:18.711990 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:18.711543    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl" podUID="289be72d-8c4c-4188-b86e-ce5dfd88ae92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 10 17:39:19.711020 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:19.710973    2552 patch_prober.go:28] interesting pod/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 10 17:39:19.711241 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:19.711044    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl" podUID="289be72d-8c4c-4188-b86e-ce5dfd88ae92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 10 17:39:20.200755 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:20.200710    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:39:20.711392 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:20.711350    2552 patch_prober.go:28] interesting pod/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 10 17:39:20.711623 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:20.711426    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl" podUID="289be72d-8c4c-4188-b86e-ce5dfd88ae92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 10 17:39:21.714460 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:21.714427    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:39:21.714936 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:21.714833    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:39:21.715519 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:21.715485    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:39:24.822515 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:24.822453    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:24.822515 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:24.822519    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:24.834703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:24.834676    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:25.462964 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:25.462927    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:39:36.603083 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:36.603048    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:39:38.601980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.601945    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"]
Sep 10 17:39:38.605114 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.605091    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.609056 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.609029    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-14b3e7\"/\"default-dockercfg-6gp52\""
Sep 10 17:39:38.609190 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.609030    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-14b3e7\"/\"explicit-model-14b3e7-kserve-self-signed-certs\""
Sep 10 17:39:38.609190 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.609029    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-14b3e7\"/\"kube-root-ca.crt\""
Sep 10 17:39:38.609190 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.609029    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-14b3e7\"/\"openshift-service-ca.crt\""
Sep 10 17:39:38.618159 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.618133    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"]
Sep 10 17:39:38.657332 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.657292    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-dshm\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.657530 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.657410    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8994c0da-cf6d-4aa8-8523-26bd21836a68-tls-certs\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.657530 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.657486    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-tmp-dir\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.657671 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.657558    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-model-cache\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.657671 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.657608    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfnf8\" (UniqueName: \"kubernetes.io/projected/8994c0da-cf6d-4aa8-8523-26bd21836a68-kube-api-access-cfnf8\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.657671 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.657652    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-home\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.759082 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.759044    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-tmp-dir\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.759283 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.759091    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-model-cache\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.759283 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.759125    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfnf8\" (UniqueName: \"kubernetes.io/projected/8994c0da-cf6d-4aa8-8523-26bd21836a68-kube-api-access-cfnf8\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.759283 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.759160    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-home\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.759283 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.759223    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-dshm\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.759531 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.759290    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8994c0da-cf6d-4aa8-8523-26bd21836a68-tls-certs\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.759600 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.759536    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-tmp-dir\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.759656 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.759609    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-model-cache\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.759699 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.759659    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-home\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.761628 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.761603    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-dshm\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.761900 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.761880    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8994c0da-cf6d-4aa8-8523-26bd21836a68-tls-certs\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.767138 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.767116    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfnf8\" (UniqueName: \"kubernetes.io/projected/8994c0da-cf6d-4aa8-8523-26bd21836a68-kube-api-access-cfnf8\") pod \"explicit-model-14b3e7-kserve-7c57967bf-r8mnk\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") " pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:38.914544 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:38.914441    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:39.186948 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:39.186831    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"]
Sep 10 17:39:39.189308 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:39:39.189280    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8994c0da_cf6d_4aa8_8523_26bd21836a68.slice/crio-76f3503756c85cd24b658d98f2268c8cb01f5f9b22a8f4afe03b16002664f6f1 WatchSource:0}: Error finding container 76f3503756c85cd24b658d98f2268c8cb01f5f9b22a8f4afe03b16002664f6f1: Status 404 returned error can't find the container with id 76f3503756c85cd24b658d98f2268c8cb01f5f9b22a8f4afe03b16002664f6f1
Sep 10 17:39:39.526627 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:39.526519    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk" event={"ID":"8994c0da-cf6d-4aa8-8523-26bd21836a68","Type":"ContainerStarted","Data":"5b329c49158b390230f8d49850a19d26316d0b7d5b31a3e1eccebc46b5f88330"}
Sep 10 17:39:39.526627 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:39.526574    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk" event={"ID":"8994c0da-cf6d-4aa8-8523-26bd21836a68","Type":"ContainerStarted","Data":"76f3503756c85cd24b658d98f2268c8cb01f5f9b22a8f4afe03b16002664f6f1"}
Sep 10 17:39:39.549766 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:39.549710    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk" podStartSLOduration=1.5496944259999998 podStartE2EDuration="1.549694426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:39:39.547291869 +0000 UTC m=+951.305818938" watchObservedRunningTime="2026-09-10 17:39:39.549694426 +0000 UTC m=+951.308221496"
Sep 10 17:39:48.914593 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:48.914565    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:48.915095 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:48.914607    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:48.927963 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:48.927933    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:49.590966 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:49.590934    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:39:52.080723 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:52.080687    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"]
Sep 10 17:39:53.096414 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:39:53.096375    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:40:04.309432 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:04.309396    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:40:07.100426 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.100387    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk" podUID="8994c0da-cf6d-4aa8-8523-26bd21836a68" containerName="main" containerID="cri-o://5b329c49158b390230f8d49850a19d26316d0b7d5b31a3e1eccebc46b5f88330" gracePeriod=45
Sep 10 17:40:07.353942 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.353875    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:40:07.458198 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.458157    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8994c0da-cf6d-4aa8-8523-26bd21836a68-tls-certs\") pod \"8994c0da-cf6d-4aa8-8523-26bd21836a68\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") "
Sep 10 17:40:07.458198 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.458208    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-home\") pod \"8994c0da-cf6d-4aa8-8523-26bd21836a68\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") "
Sep 10 17:40:07.458408 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.458252    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfnf8\" (UniqueName: \"kubernetes.io/projected/8994c0da-cf6d-4aa8-8523-26bd21836a68-kube-api-access-cfnf8\") pod \"8994c0da-cf6d-4aa8-8523-26bd21836a68\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") "
Sep 10 17:40:07.458408 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.458283    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-tmp-dir\") pod \"8994c0da-cf6d-4aa8-8523-26bd21836a68\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") "
Sep 10 17:40:07.458408 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.458306    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-dshm\") pod \"8994c0da-cf6d-4aa8-8523-26bd21836a68\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") "
Sep 10 17:40:07.458408 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.458381    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-model-cache\") pod \"8994c0da-cf6d-4aa8-8523-26bd21836a68\" (UID: \"8994c0da-cf6d-4aa8-8523-26bd21836a68\") "
Sep 10 17:40:07.458630 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.458533    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-home" (OuterVolumeSpecName: "home") pod "8994c0da-cf6d-4aa8-8523-26bd21836a68" (UID: "8994c0da-cf6d-4aa8-8523-26bd21836a68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:07.458770 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.458737    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8994c0da-cf6d-4aa8-8523-26bd21836a68" (UID: "8994c0da-cf6d-4aa8-8523-26bd21836a68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:07.458770 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.458758    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-model-cache" (OuterVolumeSpecName: "model-cache") pod "8994c0da-cf6d-4aa8-8523-26bd21836a68" (UID: "8994c0da-cf6d-4aa8-8523-26bd21836a68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:07.458945 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.458782    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-home\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:40:07.460590 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.460558    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-dshm" (OuterVolumeSpecName: "dshm") pod "8994c0da-cf6d-4aa8-8523-26bd21836a68" (UID: "8994c0da-cf6d-4aa8-8523-26bd21836a68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:07.460679 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.460586    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8994c0da-cf6d-4aa8-8523-26bd21836a68-kube-api-access-cfnf8" (OuterVolumeSpecName: "kube-api-access-cfnf8") pod "8994c0da-cf6d-4aa8-8523-26bd21836a68" (UID: "8994c0da-cf6d-4aa8-8523-26bd21836a68"). InnerVolumeSpecName "kube-api-access-cfnf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:40:07.460722 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.460690    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8994c0da-cf6d-4aa8-8523-26bd21836a68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8994c0da-cf6d-4aa8-8523-26bd21836a68" (UID: "8994c0da-cf6d-4aa8-8523-26bd21836a68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:40:07.559996 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.559962    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8994c0da-cf6d-4aa8-8523-26bd21836a68-tls-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:40:07.559996 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.559991    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfnf8\" (UniqueName: \"kubernetes.io/projected/8994c0da-cf6d-4aa8-8523-26bd21836a68-kube-api-access-cfnf8\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:40:07.559996 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.560001    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-tmp-dir\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:40:07.560230 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.560010    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-dshm\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:40:07.560230 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.560018    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8994c0da-cf6d-4aa8-8523-26bd21836a68-model-cache\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:40:07.675064 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.674979    2552 generic.go:356] "Generic (PLEG): container finished" podID="8994c0da-cf6d-4aa8-8523-26bd21836a68" containerID="5b329c49158b390230f8d49850a19d26316d0b7d5b31a3e1eccebc46b5f88330" exitCode=0
Sep 10 17:40:07.675064 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.675018    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"
Sep 10 17:40:07.675285 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.675066    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk" event={"ID":"8994c0da-cf6d-4aa8-8523-26bd21836a68","Type":"ContainerDied","Data":"5b329c49158b390230f8d49850a19d26316d0b7d5b31a3e1eccebc46b5f88330"}
Sep 10 17:40:07.675285 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.675111    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk" event={"ID":"8994c0da-cf6d-4aa8-8523-26bd21836a68","Type":"ContainerDied","Data":"76f3503756c85cd24b658d98f2268c8cb01f5f9b22a8f4afe03b16002664f6f1"}
Sep 10 17:40:07.675285 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.675133    2552 scope.go:117] "RemoveContainer" containerID="5b329c49158b390230f8d49850a19d26316d0b7d5b31a3e1eccebc46b5f88330"
Sep 10 17:40:07.684325 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.684306    2552 scope.go:117] "RemoveContainer" containerID="5b329c49158b390230f8d49850a19d26316d0b7d5b31a3e1eccebc46b5f88330"
Sep 10 17:40:07.684640 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:40:07.684617    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b329c49158b390230f8d49850a19d26316d0b7d5b31a3e1eccebc46b5f88330\": container with ID starting with 5b329c49158b390230f8d49850a19d26316d0b7d5b31a3e1eccebc46b5f88330 not found: ID does not exist" containerID="5b329c49158b390230f8d49850a19d26316d0b7d5b31a3e1eccebc46b5f88330"
Sep 10 17:40:07.684732 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.684647    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b329c49158b390230f8d49850a19d26316d0b7d5b31a3e1eccebc46b5f88330"} err="failed to get container status \"5b329c49158b390230f8d49850a19d26316d0b7d5b31a3e1eccebc46b5f88330\": rpc error: code = NotFound desc = could not find container \"5b329c49158b390230f8d49850a19d26316d0b7d5b31a3e1eccebc46b5f88330\": container with ID starting with 5b329c49158b390230f8d49850a19d26316d0b7d5b31a3e1eccebc46b5f88330 not found: ID does not exist"
Sep 10 17:40:07.695576 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.695552    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"]
Sep 10 17:40:07.699338 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:07.699315    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-14b3e7/explicit-model-14b3e7-kserve-7c57967bf-r8mnk"]
Sep 10 17:40:08.803422 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:08.803385    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8994c0da-cf6d-4aa8-8523-26bd21836a68" path="/var/lib/kubelet/pods/8994c0da-cf6d-4aa8-8523-26bd21836a68/volumes"
Sep 10 17:40:09.394418 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:09.394382    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"]
Sep 10 17:40:09.394655 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:09.394616    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl" podUID="289be72d-8c4c-4188-b86e-ce5dfd88ae92" containerName="istio-proxy" containerID="cri-o://c4f5d96670783dde3090a413660374ed62e8aefb6d2111c071bbae5c36bdee7a" gracePeriod=30
Sep 10 17:40:14.654296 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.654266    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:40:14.713701 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.713598    2552 generic.go:356] "Generic (PLEG): container finished" podID="289be72d-8c4c-4188-b86e-ce5dfd88ae92" containerID="c4f5d96670783dde3090a413660374ed62e8aefb6d2111c071bbae5c36bdee7a" exitCode=0
Sep 10 17:40:14.713701 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.713655    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"
Sep 10 17:40:14.713701 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.713685    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl" event={"ID":"289be72d-8c4c-4188-b86e-ce5dfd88ae92","Type":"ContainerDied","Data":"c4f5d96670783dde3090a413660374ed62e8aefb6d2111c071bbae5c36bdee7a"}
Sep 10 17:40:14.713972 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.713774    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl" event={"ID":"289be72d-8c4c-4188-b86e-ce5dfd88ae92","Type":"ContainerDied","Data":"9dc098b55fa23b99a0064efeb689e909689d1cda9222a6d6d739beb171c678ae"}
Sep 10 17:40:14.713972 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.713799    2552 scope.go:117] "RemoveContainer" containerID="c4f5d96670783dde3090a413660374ed62e8aefb6d2111c071bbae5c36bdee7a"
Sep 10 17:40:14.723022 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.723004    2552 scope.go:117] "RemoveContainer" containerID="f797d22925bb14df253ea6ca598bb8c2c1b1ecf02ea21d92801a68b4b240b1f2"
Sep 10 17:40:14.730905 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.730891    2552 scope.go:117] "RemoveContainer" containerID="c4f5d96670783dde3090a413660374ed62e8aefb6d2111c071bbae5c36bdee7a"
Sep 10 17:40:14.731185 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:40:14.731162    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f5d96670783dde3090a413660374ed62e8aefb6d2111c071bbae5c36bdee7a\": container with ID starting with c4f5d96670783dde3090a413660374ed62e8aefb6d2111c071bbae5c36bdee7a not found: ID does not exist" containerID="c4f5d96670783dde3090a413660374ed62e8aefb6d2111c071bbae5c36bdee7a"
Sep 10 17:40:14.731278 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.731196    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f5d96670783dde3090a413660374ed62e8aefb6d2111c071bbae5c36bdee7a"} err="failed to get container status \"c4f5d96670783dde3090a413660374ed62e8aefb6d2111c071bbae5c36bdee7a\": rpc error: code = NotFound desc = could not find container \"c4f5d96670783dde3090a413660374ed62e8aefb6d2111c071bbae5c36bdee7a\": container with ID starting with c4f5d96670783dde3090a413660374ed62e8aefb6d2111c071bbae5c36bdee7a not found: ID does not exist"
Sep 10 17:40:14.731278 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.731220    2552 scope.go:117] "RemoveContainer" containerID="f797d22925bb14df253ea6ca598bb8c2c1b1ecf02ea21d92801a68b4b240b1f2"
Sep 10 17:40:14.731278 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.731166    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-podinfo\") pod \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") "
Sep 10 17:40:14.731451 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.731333    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-envoy\") pod \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") "
Sep 10 17:40:14.731451 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.731398    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-workload-certs\") pod \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") "
Sep 10 17:40:14.731451 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.731442    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5h2l\" (UniqueName: \"kubernetes.io/projected/289be72d-8c4c-4188-b86e-ce5dfd88ae92-kube-api-access-m5h2l\") pod \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") "
Sep 10 17:40:14.731632 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:40:14.731557    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f797d22925bb14df253ea6ca598bb8c2c1b1ecf02ea21d92801a68b4b240b1f2\": container with ID starting with f797d22925bb14df253ea6ca598bb8c2c1b1ecf02ea21d92801a68b4b240b1f2 not found: ID does not exist" containerID="f797d22925bb14df253ea6ca598bb8c2c1b1ecf02ea21d92801a68b4b240b1f2"
Sep 10 17:40:14.731632 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.731592    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f797d22925bb14df253ea6ca598bb8c2c1b1ecf02ea21d92801a68b4b240b1f2"} err="failed to get container status \"f797d22925bb14df253ea6ca598bb8c2c1b1ecf02ea21d92801a68b4b240b1f2\": rpc error: code = NotFound desc = could not find container \"f797d22925bb14df253ea6ca598bb8c2c1b1ecf02ea21d92801a68b4b240b1f2\": container with ID starting with f797d22925bb14df253ea6ca598bb8c2c1b1ecf02ea21d92801a68b4b240b1f2 not found: ID does not exist"
Sep 10 17:40:14.731746 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.731650    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istiod-ca-cert\") pod \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") "
Sep 10 17:40:14.731746 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.731708    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "289be72d-8c4c-4188-b86e-ce5dfd88ae92" (UID: "289be72d-8c4c-4188-b86e-ce5dfd88ae92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:14.731746 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.731731    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-workload-socket\") pod \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") "
Sep 10 17:40:14.731897 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.731756    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-data\") pod \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") "
Sep 10 17:40:14.731897 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.731791    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-token\") pod \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") "
Sep 10 17:40:14.731897 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.731826    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-credential-socket\") pod \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\" (UID: \"289be72d-8c4c-4188-b86e-ce5dfd88ae92\") "
Sep 10 17:40:14.732213 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.732107    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-data" (OuterVolumeSpecName: "istio-data") pod "289be72d-8c4c-4188-b86e-ce5dfd88ae92" (UID: "289be72d-8c4c-4188-b86e-ce5dfd88ae92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:14.732213 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.732169    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-workload-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:40:14.732213 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.732185    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "289be72d-8c4c-4188-b86e-ce5dfd88ae92" (UID: "289be72d-8c4c-4188-b86e-ce5dfd88ae92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:14.732393 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.732299    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "289be72d-8c4c-4188-b86e-ce5dfd88ae92" (UID: "289be72d-8c4c-4188-b86e-ce5dfd88ae92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:14.732512 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.732472    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "289be72d-8c4c-4188-b86e-ce5dfd88ae92" (UID: "289be72d-8c4c-4188-b86e-ce5dfd88ae92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:40:14.733347 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.733325    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "289be72d-8c4c-4188-b86e-ce5dfd88ae92" (UID: "289be72d-8c4c-4188-b86e-ce5dfd88ae92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:40:14.733576 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.733559    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "289be72d-8c4c-4188-b86e-ce5dfd88ae92" (UID: "289be72d-8c4c-4188-b86e-ce5dfd88ae92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:14.734005 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.733974    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289be72d-8c4c-4188-b86e-ce5dfd88ae92-kube-api-access-m5h2l" (OuterVolumeSpecName: "kube-api-access-m5h2l") pod "289be72d-8c4c-4188-b86e-ce5dfd88ae92" (UID: "289be72d-8c4c-4188-b86e-ce5dfd88ae92"). InnerVolumeSpecName "kube-api-access-m5h2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:40:14.734091 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.734029    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-token" (OuterVolumeSpecName: "istio-token") pod "289be72d-8c4c-4188-b86e-ce5dfd88ae92" (UID: "289be72d-8c4c-4188-b86e-ce5dfd88ae92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:40:14.833431 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.833395    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5h2l\" (UniqueName: \"kubernetes.io/projected/289be72d-8c4c-4188-b86e-ce5dfd88ae92-kube-api-access-m5h2l\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:40:14.833431 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.833425    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istiod-ca-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:40:14.833431 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.833435    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-workload-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:40:14.833685 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.833448    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-data\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:40:14.833685 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.833457    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-token\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:40:14.833685 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.833465    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-credential-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:40:14.833685 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.833473    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-podinfo\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:40:14.833685 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:14.833481    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289be72d-8c4c-4188-b86e-ce5dfd88ae92-istio-envoy\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:40:15.031087 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:15.030995    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"]
Sep 10 17:40:15.034927 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:15.034897    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-14b3e7-openshift-default-7784fccbb4-sdhvl"]
Sep 10 17:40:15.700027 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:15.699984    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:40:16.803790 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:16.803747    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="289be72d-8c4c-4188-b86e-ce5dfd88ae92" path="/var/lib/kubelet/pods/289be72d-8c4c-4188-b86e-ce5dfd88ae92/volumes"
Sep 10 17:40:28.097906 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:28.097865    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:40:30.306389 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.306353    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"]
Sep 10 17:40:30.306992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.306970    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289be72d-8c4c-4188-b86e-ce5dfd88ae92" containerName="istio-proxy"
Sep 10 17:40:30.307103 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.306994    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="289be72d-8c4c-4188-b86e-ce5dfd88ae92" containerName="istio-proxy"
Sep 10 17:40:30.307103 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.307007    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8994c0da-cf6d-4aa8-8523-26bd21836a68" containerName="main"
Sep 10 17:40:30.307103 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.307016    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8994c0da-cf6d-4aa8-8523-26bd21836a68" containerName="main"
Sep 10 17:40:30.307103 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.307035    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289be72d-8c4c-4188-b86e-ce5dfd88ae92" containerName="istio-proxy"
Sep 10 17:40:30.307103 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.307046    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="289be72d-8c4c-4188-b86e-ce5dfd88ae92" containerName="istio-proxy"
Sep 10 17:40:30.307358 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.307172    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8994c0da-cf6d-4aa8-8523-26bd21836a68" containerName="main"
Sep 10 17:40:30.307358 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.307186    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="289be72d-8c4c-4188-b86e-ce5dfd88ae92" containerName="istio-proxy"
Sep 10 17:40:30.310553 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.310531    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.313324 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.313301    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-aee2c26d-openshift-default-dockercfg-nhqmr\""
Sep 10 17:40:30.327904 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.327878    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"]
Sep 10 17:40:30.483597 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.483549    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-credential-socket\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.483597 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.483597    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-envoy\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.483827 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.483631    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istiod-ca-cert\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.483827 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.483669    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-token\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.483827 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.483689    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-workload-certs\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.483827 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.483765    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxxvg\" (UniqueName: \"kubernetes.io/projected/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-kube-api-access-dxxvg\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.483827 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.483813    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-podinfo\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.484006 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.483892    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-workload-socket\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.484006 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.483916    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-data\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.585311 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.585275    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-token\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.585525 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.585321    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-workload-certs\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.585525 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.585360    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxxvg\" (UniqueName: \"kubernetes.io/projected/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-kube-api-access-dxxvg\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.585525 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.585387    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-podinfo\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.585525 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.585425    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-workload-socket\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.585525 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.585449    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-data\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.585525 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.585485    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-credential-socket\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.585871 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.585550    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-envoy\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.585871 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.585599    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istiod-ca-cert\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.585980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.585882    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-workload-socket\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.585980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.585897    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-workload-certs\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.585980 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.585964    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-data\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.586130 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.586028    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-credential-socket\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.586297 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.586276    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istiod-ca-cert\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.587906 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.587885    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-envoy\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.588036 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.588016    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-podinfo\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.593787 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.593757    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-token\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.594239 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.594216    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxxvg\" (UniqueName: \"kubernetes.io/projected/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-kube-api-access-dxxvg\") pod \"e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") " pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.621288 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.621251    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:30.899293 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.899267    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"]
Sep 10 17:40:30.901434 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:40:30.901402    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4b9288_cfb9_4afc_aa75_f85d7f61c66b.slice/crio-2a2710f61e920b6b6092cea482bcb70279d6e0f365603ace0bea52dec50e85a0 WatchSource:0}: Error finding container 2a2710f61e920b6b6092cea482bcb70279d6e0f365603ace0bea52dec50e85a0: Status 404 returned error can't find the container with id 2a2710f61e920b6b6092cea482bcb70279d6e0f365603ace0bea52dec50e85a0
Sep 10 17:40:30.903775 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.903738    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:40:30.903875 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.903811    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:40:30.903875 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:30.903840    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:40:31.808033 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:31.807987    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn" event={"ID":"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b","Type":"ContainerStarted","Data":"5623cc1da56316728709a8f5283f8e2ba51669c4f3f1c4f7fdca70420a003a14"}
Sep 10 17:40:31.808592 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:31.808056    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn" event={"ID":"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b","Type":"ContainerStarted","Data":"2a2710f61e920b6b6092cea482bcb70279d6e0f365603ace0bea52dec50e85a0"}
Sep 10 17:40:31.834378 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:31.834323    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn" podStartSLOduration=1.834306775 podStartE2EDuration="1.834306775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:40:31.831879776 +0000 UTC m=+1003.590406846" watchObservedRunningTime="2026-09-10 17:40:31.834306775 +0000 UTC m=+1003.592833845"
Sep 10 17:40:32.622048 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:32.622004    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:32.627075 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:32.627047    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:32.813003 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:32.812968    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:32.814115 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:32.814093    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:42.311826 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:42.311790    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:40:51.914822 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:51.914745    2552 generic.go:356] "Generic (PLEG): container finished" podID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" containerID="5623cc1da56316728709a8f5283f8e2ba51669c4f3f1c4f7fdca70420a003a14" exitCode=0
Sep 10 17:40:51.914822 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:51.914792    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn" event={"ID":"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b","Type":"ContainerDied","Data":"5623cc1da56316728709a8f5283f8e2ba51669c4f3f1c4f7fdca70420a003a14"}
Sep 10 17:40:51.915245 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:51.915190    2552 scope.go:117] "RemoveContainer" containerID="5623cc1da56316728709a8f5283f8e2ba51669c4f3f1c4f7fdca70420a003a14"
Sep 10 17:40:51.916348 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:51.916318    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:40:51.916409 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:51.916387    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:40:51.916446 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:51.916416    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:40:52.921766 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:52.921727    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn" event={"ID":"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b","Type":"ContainerStarted","Data":"08aaa77a397442069b91c33a506b4ea193dc58db6c0e7c806f0d6616cf6ad5c2"}
Sep 10 17:40:53.622113 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:53.622077    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:53.622294 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:53.622261    2552 patch_prober.go:28] interesting pod/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 10 17:40:53.622345 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:53.622310    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn" podUID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 10 17:40:53.796076 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:53.796033    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:40:54.621853 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:54.621813    2552 patch_prober.go:28] interesting pod/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 10 17:40:54.622241 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:54.621877    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn" podUID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 10 17:40:55.622302 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:55.622265    2552 patch_prober.go:28] interesting pod/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 10 17:40:55.622722 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:55.622325    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn" podUID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 10 17:40:56.622133 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:56.622089    2552 patch_prober.go:28] interesting pod/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 10 17:40:56.622337 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:56.622144    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn" podUID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 10 17:40:57.622178 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:57.622135    2552 patch_prober.go:28] interesting pod/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 10 17:40:57.622435 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:57.622195    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn" podUID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 10 17:40:58.622144 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:58.622107    2552 patch_prober.go:28] interesting pod/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 10 17:40:58.622324 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:58.622179    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn" podUID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 10 17:40:59.625539 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:59.625501    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:59.625927 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:59.625773    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:40:59.626596 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:40:59.626575    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:41:01.986434 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:01.986399    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"]
Sep 10 17:41:02.890777 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:02.890743    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"]
Sep 10 17:41:05.403460 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:05.403421    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:41:16.405487 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:16.405451    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:41:17.006562 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.006487    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh" podUID="3d772bb2-787c-439a-8e3c-c082df619445" containerName="main" containerID="cri-o://75b194483746c138e46dd4e54efa6224fb00d12f24cb8004934a6f8421e95ee6" gracePeriod=45
Sep 10 17:41:17.249457 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.249429    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:41:17.352838 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.352809    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-tmp-dir\") pod \"3d772bb2-787c-439a-8e3c-c082df619445\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") "
Sep 10 17:41:17.353020 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.352856    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-model-cache\") pod \"3d772bb2-787c-439a-8e3c-c082df619445\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") "
Sep 10 17:41:17.353020 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.352904    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d772bb2-787c-439a-8e3c-c082df619445-tls-certs\") pod \"3d772bb2-787c-439a-8e3c-c082df619445\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") "
Sep 10 17:41:17.353020 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.352970    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-home\") pod \"3d772bb2-787c-439a-8e3c-c082df619445\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") "
Sep 10 17:41:17.353020 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.353012    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-dshm\") pod \"3d772bb2-787c-439a-8e3c-c082df619445\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") "
Sep 10 17:41:17.353252 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.353038    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6sq5\" (UniqueName: \"kubernetes.io/projected/3d772bb2-787c-439a-8e3c-c082df619445-kube-api-access-z6sq5\") pod \"3d772bb2-787c-439a-8e3c-c082df619445\" (UID: \"3d772bb2-787c-439a-8e3c-c082df619445\") "
Sep 10 17:41:17.353252 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.353100    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d772bb2-787c-439a-8e3c-c082df619445" (UID: "3d772bb2-787c-439a-8e3c-c082df619445"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:17.353477 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.353459    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-tmp-dir\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:17.353645 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.353577    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-home" (OuterVolumeSpecName: "home") pod "3d772bb2-787c-439a-8e3c-c082df619445" (UID: "3d772bb2-787c-439a-8e3c-c082df619445"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:17.353778 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.353725    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d772bb2-787c-439a-8e3c-c082df619445" (UID: "3d772bb2-787c-439a-8e3c-c082df619445"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:17.356191 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.356152    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d772bb2-787c-439a-8e3c-c082df619445-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d772bb2-787c-439a-8e3c-c082df619445" (UID: "3d772bb2-787c-439a-8e3c-c082df619445"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:41:17.356580 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.356403    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d772bb2-787c-439a-8e3c-c082df619445-kube-api-access-z6sq5" (OuterVolumeSpecName: "kube-api-access-z6sq5") pod "3d772bb2-787c-439a-8e3c-c082df619445" (UID: "3d772bb2-787c-439a-8e3c-c082df619445"). InnerVolumeSpecName "kube-api-access-z6sq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:41:17.356829 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.356807    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-dshm" (OuterVolumeSpecName: "dshm") pod "3d772bb2-787c-439a-8e3c-c082df619445" (UID: "3d772bb2-787c-439a-8e3c-c082df619445"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:17.454855 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.454815    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-model-cache\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:17.454855 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.454848    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d772bb2-787c-439a-8e3c-c082df619445-tls-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:17.454855 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.454856    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-home\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:17.454855 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.454863    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d772bb2-787c-439a-8e3c-c082df619445-dshm\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:17.455426 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.454872    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6sq5\" (UniqueName: \"kubernetes.io/projected/3d772bb2-787c-439a-8e3c-c082df619445-kube-api-access-z6sq5\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:17.910851 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:17.910811    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm" podUID="67945228-9082-4a56-921c-516aec05cfcf" containerName="main" containerID="cri-o://ed0644c1c63dc7df6f0ecbc7308d3179a50baf35580713c62819c267e4e40f76" gracePeriod=45
Sep 10 17:41:18.061270 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.061238    2552 generic.go:356] "Generic (PLEG): container finished" podID="67945228-9082-4a56-921c-516aec05cfcf" containerID="ed0644c1c63dc7df6f0ecbc7308d3179a50baf35580713c62819c267e4e40f76" exitCode=0
Sep 10 17:41:18.061452 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.061314    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm" event={"ID":"67945228-9082-4a56-921c-516aec05cfcf","Type":"ContainerDied","Data":"ed0644c1c63dc7df6f0ecbc7308d3179a50baf35580713c62819c267e4e40f76"}
Sep 10 17:41:18.062719 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.062681    2552 generic.go:356] "Generic (PLEG): container finished" podID="3d772bb2-787c-439a-8e3c-c082df619445" containerID="75b194483746c138e46dd4e54efa6224fb00d12f24cb8004934a6f8421e95ee6" exitCode=0
Sep 10 17:41:18.062829 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.062724    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"
Sep 10 17:41:18.062829 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.062754    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh" event={"ID":"3d772bb2-787c-439a-8e3c-c082df619445","Type":"ContainerDied","Data":"75b194483746c138e46dd4e54efa6224fb00d12f24cb8004934a6f8421e95ee6"}
Sep 10 17:41:18.062829 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.062794    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh" event={"ID":"3d772bb2-787c-439a-8e3c-c082df619445","Type":"ContainerDied","Data":"e2950dc546244e17a4acdd0c3184824249050c1f5577394d33e59d7c6f1918c0"}
Sep 10 17:41:18.062829 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.062816    2552 scope.go:117] "RemoveContainer" containerID="75b194483746c138e46dd4e54efa6224fb00d12f24cb8004934a6f8421e95ee6"
Sep 10 17:41:18.072840 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.072824    2552 scope.go:117] "RemoveContainer" containerID="75b194483746c138e46dd4e54efa6224fb00d12f24cb8004934a6f8421e95ee6"
Sep 10 17:41:18.073131 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:41:18.073111    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b194483746c138e46dd4e54efa6224fb00d12f24cb8004934a6f8421e95ee6\": container with ID starting with 75b194483746c138e46dd4e54efa6224fb00d12f24cb8004934a6f8421e95ee6 not found: ID does not exist" containerID="75b194483746c138e46dd4e54efa6224fb00d12f24cb8004934a6f8421e95ee6"
Sep 10 17:41:18.073195 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.073144    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b194483746c138e46dd4e54efa6224fb00d12f24cb8004934a6f8421e95ee6"} err="failed to get container status \"75b194483746c138e46dd4e54efa6224fb00d12f24cb8004934a6f8421e95ee6\": rpc error: code = NotFound desc = could not find container \"75b194483746c138e46dd4e54efa6224fb00d12f24cb8004934a6f8421e95ee6\": container with ID starting with 75b194483746c138e46dd4e54efa6224fb00d12f24cb8004934a6f8421e95ee6 not found: ID does not exist"
Sep 10 17:41:18.086621 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.086596    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"]
Sep 10 17:41:18.094598 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.094558    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0d25fa/test-model-0d25fa-kserve-545657d6df-vwzsh"]
Sep 10 17:41:18.163576 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.163513    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:41:18.262280 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.262247    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-home\") pod \"67945228-9082-4a56-921c-516aec05cfcf\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") "
Sep 10 17:41:18.262438 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.262291    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-tmp-dir\") pod \"67945228-9082-4a56-921c-516aec05cfcf\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") "
Sep 10 17:41:18.262438 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.262326    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-dshm\") pod \"67945228-9082-4a56-921c-516aec05cfcf\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") "
Sep 10 17:41:18.262438 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.262406    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45zxm\" (UniqueName: \"kubernetes.io/projected/67945228-9082-4a56-921c-516aec05cfcf-kube-api-access-45zxm\") pod \"67945228-9082-4a56-921c-516aec05cfcf\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") "
Sep 10 17:41:18.262438 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.262424    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-model-cache\") pod \"67945228-9082-4a56-921c-516aec05cfcf\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") "
Sep 10 17:41:18.262685 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.262514    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67945228-9082-4a56-921c-516aec05cfcf-tls-certs\") pod \"67945228-9082-4a56-921c-516aec05cfcf\" (UID: \"67945228-9082-4a56-921c-516aec05cfcf\") "
Sep 10 17:41:18.262685 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.262525    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-home" (OuterVolumeSpecName: "home") pod "67945228-9082-4a56-921c-516aec05cfcf" (UID: "67945228-9082-4a56-921c-516aec05cfcf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:18.262685 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.262626    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67945228-9082-4a56-921c-516aec05cfcf" (UID: "67945228-9082-4a56-921c-516aec05cfcf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:18.262851 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.262773    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-model-cache" (OuterVolumeSpecName: "model-cache") pod "67945228-9082-4a56-921c-516aec05cfcf" (UID: "67945228-9082-4a56-921c-516aec05cfcf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:18.262911 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.262884    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-home\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:18.262911 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.262902    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-tmp-dir\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:18.262991 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.262915    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-model-cache\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:18.264642 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.264616    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-dshm" (OuterVolumeSpecName: "dshm") pod "67945228-9082-4a56-921c-516aec05cfcf" (UID: "67945228-9082-4a56-921c-516aec05cfcf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:18.264761 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.264660    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67945228-9082-4a56-921c-516aec05cfcf-kube-api-access-45zxm" (OuterVolumeSpecName: "kube-api-access-45zxm") pod "67945228-9082-4a56-921c-516aec05cfcf" (UID: "67945228-9082-4a56-921c-516aec05cfcf"). InnerVolumeSpecName "kube-api-access-45zxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:41:18.264761 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.264693    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67945228-9082-4a56-921c-516aec05cfcf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67945228-9082-4a56-921c-516aec05cfcf" (UID: "67945228-9082-4a56-921c-516aec05cfcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:41:18.363743 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.363705    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45zxm\" (UniqueName: \"kubernetes.io/projected/67945228-9082-4a56-921c-516aec05cfcf-kube-api-access-45zxm\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:18.363743 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.363737    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67945228-9082-4a56-921c-516aec05cfcf-tls-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:18.363743 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.363749    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67945228-9082-4a56-921c-516aec05cfcf-dshm\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:18.803761 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:18.803731    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d772bb2-787c-439a-8e3c-c082df619445" path="/var/lib/kubelet/pods/3d772bb2-787c-439a-8e3c-c082df619445/volumes"
Sep 10 17:41:19.069133 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:19.069040    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"
Sep 10 17:41:19.069133 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:19.069039    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm" event={"ID":"67945228-9082-4a56-921c-516aec05cfcf","Type":"ContainerDied","Data":"c932ec3ddb16019f341d962bdcba09c2d04682acc3d1fc60ea34e83652d14a0e"}
Sep 10 17:41:19.069343 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:19.069184    2552 scope.go:117] "RemoveContainer" containerID="ed0644c1c63dc7df6f0ecbc7308d3179a50baf35580713c62819c267e4e40f76"
Sep 10 17:41:19.085758 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:19.085734    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"]
Sep 10 17:41:19.093408 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:19.093382    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2f1093/test-model-2f1093-kserve-6b8969dd75-9spnm"]
Sep 10 17:41:20.803385 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:20.803351    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67945228-9082-4a56-921c-516aec05cfcf" path="/var/lib/kubelet/pods/67945228-9082-4a56-921c-516aec05cfcf/volumes"
Sep 10 17:41:23.973415 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:23.973384    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"]
Sep 10 17:41:23.973880 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:23.973617    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn" podUID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" containerName="istio-proxy" containerID="cri-o://08aaa77a397442069b91c33a506b4ea193dc58db6c0e7c806f0d6616cf6ad5c2" gracePeriod=30
Sep 10 17:41:27.895114 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:27.895077    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:41:29.124486 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.124451    2552 generic.go:356] "Generic (PLEG): container finished" podID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" containerID="08aaa77a397442069b91c33a506b4ea193dc58db6c0e7c806f0d6616cf6ad5c2" exitCode=0
Sep 10 17:41:29.124905 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.124530    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn" event={"ID":"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b","Type":"ContainerDied","Data":"08aaa77a397442069b91c33a506b4ea193dc58db6c0e7c806f0d6616cf6ad5c2"}
Sep 10 17:41:29.124905 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.124607    2552 scope.go:117] "RemoveContainer" containerID="5623cc1da56316728709a8f5283f8e2ba51669c4f3f1c4f7fdca70420a003a14"
Sep 10 17:41:29.219277 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.219250    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:41:29.373204 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.373165    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-envoy\") pod \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") "
Sep 10 17:41:29.373401 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.373213    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-podinfo\") pod \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") "
Sep 10 17:41:29.373401 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.373249    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-token\") pod \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") "
Sep 10 17:41:29.373401 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.373280    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxxvg\" (UniqueName: \"kubernetes.io/projected/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-kube-api-access-dxxvg\") pod \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") "
Sep 10 17:41:29.373401 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.373333    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-credential-socket\") pod \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") "
Sep 10 17:41:29.373401 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.373363    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-workload-socket\") pod \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") "
Sep 10 17:41:29.373401 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.373388    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istiod-ca-cert\") pod \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") "
Sep 10 17:41:29.373768 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.373426    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-workload-certs\") pod \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") "
Sep 10 17:41:29.373768 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.373588    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-data\") pod \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\" (UID: \"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b\") "
Sep 10 17:41:29.373880 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.373846    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" (UID: "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:41:29.373880 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.373856    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" (UID: "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:29.373989 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.373954    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" (UID: "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:29.373989 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.373961    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-workload-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:29.374098 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.374024    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istiod-ca-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:29.374098 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.374071    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-data" (OuterVolumeSpecName: "istio-data") pod "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" (UID: "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:29.374098 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.374084    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" (UID: "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:29.375622 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.375588    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-token" (OuterVolumeSpecName: "istio-token") pod "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" (UID: "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:41:29.375735 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.375638    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-kube-api-access-dxxvg" (OuterVolumeSpecName: "kube-api-access-dxxvg") pod "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" (UID: "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b"). InnerVolumeSpecName "kube-api-access-dxxvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:41:29.375735 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.375650    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" (UID: "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:29.375735 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.375663    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" (UID: "2c4b9288-cfb9-4afc-aa75-f85d7f61c66b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:41:29.474946 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.474907    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-data\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:29.474946 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.474940    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-envoy\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:29.474946 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.474951    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-podinfo\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:29.474946 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.474959    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-istio-token\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:29.475212 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.474968    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxxvg\" (UniqueName: \"kubernetes.io/projected/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-kube-api-access-dxxvg\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:29.475212 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.474977    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-credential-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:29.475212 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:29.474985    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b-workload-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:30.130728 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:30.130695    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"
Sep 10 17:41:30.131148 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:30.130740    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn" event={"ID":"2c4b9288-cfb9-4afc-aa75-f85d7f61c66b","Type":"ContainerDied","Data":"2a2710f61e920b6b6092cea482bcb70279d6e0f365603ace0bea52dec50e85a0"}
Sep 10 17:41:30.131148 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:30.130800    2552 scope.go:117] "RemoveContainer" containerID="08aaa77a397442069b91c33a506b4ea193dc58db6c0e7c806f0d6616cf6ad5c2"
Sep 10 17:41:30.152841 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:30.152806    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"]
Sep 10 17:41:30.157117 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:30.157091    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-aee2c26d-openshift-default-597955567b-jjhsn"]
Sep 10 17:41:30.803138 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:30.803107    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" path="/var/lib/kubelet/pods/2c4b9288-cfb9-4afc-aa75-f85d7f61c66b/volumes"
Sep 10 17:41:33.806328 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:33.806296    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:41:38.755108 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:38.755070    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"]
Sep 10 17:41:38.755589 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:38.755325    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb" podUID="71c2f787-a899-4312-b5c4-25a22cb787ae" containerName="istio-proxy" containerID="cri-o://d0ce2e14922f1e3880a31e13ca1355f29566969b48d56b697ce108ff2cbe8454" gracePeriod=30
Sep 10 17:41:44.023884 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.023809    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:41:44.122007 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.121964    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c2f787-a899-4312-b5c4-25a22cb787ae-istiod-ca-cert\") pod \"71c2f787-a899-4312-b5c4-25a22cb787ae\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") "
Sep 10 17:41:44.122216 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.122021    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4grg\" (UniqueName: \"kubernetes.io/projected/71c2f787-a899-4312-b5c4-25a22cb787ae-kube-api-access-m4grg\") pod \"71c2f787-a899-4312-b5c4-25a22cb787ae\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") "
Sep 10 17:41:44.122216 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.122060    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-workload-certs\") pod \"71c2f787-a899-4312-b5c4-25a22cb787ae\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") "
Sep 10 17:41:44.122216 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.122119    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-podinfo\") pod \"71c2f787-a899-4312-b5c4-25a22cb787ae\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") "
Sep 10 17:41:44.122216 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.122142    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-workload-socket\") pod \"71c2f787-a899-4312-b5c4-25a22cb787ae\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") "
Sep 10 17:41:44.122216 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.122189    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-token\") pod \"71c2f787-a899-4312-b5c4-25a22cb787ae\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") "
Sep 10 17:41:44.122475 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.122228    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-data\") pod \"71c2f787-a899-4312-b5c4-25a22cb787ae\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") "
Sep 10 17:41:44.122475 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.122281    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-credential-socket\") pod \"71c2f787-a899-4312-b5c4-25a22cb787ae\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") "
Sep 10 17:41:44.122475 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.122353    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-envoy\") pod \"71c2f787-a899-4312-b5c4-25a22cb787ae\" (UID: \"71c2f787-a899-4312-b5c4-25a22cb787ae\") "
Sep 10 17:41:44.122475 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.122415    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "71c2f787-a899-4312-b5c4-25a22cb787ae" (UID: "71c2f787-a899-4312-b5c4-25a22cb787ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:44.122475 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.122440    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c2f787-a899-4312-b5c4-25a22cb787ae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "71c2f787-a899-4312-b5c4-25a22cb787ae" (UID: "71c2f787-a899-4312-b5c4-25a22cb787ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:41:44.122791 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.122703    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "71c2f787-a899-4312-b5c4-25a22cb787ae" (UID: "71c2f787-a899-4312-b5c4-25a22cb787ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:44.122848 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.122820    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-credential-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:44.122848 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.122843    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c2f787-a899-4312-b5c4-25a22cb787ae-istiod-ca-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:44.122963 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.122857    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-workload-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:44.123019 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.122962    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-data" (OuterVolumeSpecName: "istio-data") pod "71c2f787-a899-4312-b5c4-25a22cb787ae" (UID: "71c2f787-a899-4312-b5c4-25a22cb787ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:44.123076 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.123023    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "71c2f787-a899-4312-b5c4-25a22cb787ae" (UID: "71c2f787-a899-4312-b5c4-25a22cb787ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:44.124540 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.124481    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c2f787-a899-4312-b5c4-25a22cb787ae-kube-api-access-m4grg" (OuterVolumeSpecName: "kube-api-access-m4grg") pod "71c2f787-a899-4312-b5c4-25a22cb787ae" (UID: "71c2f787-a899-4312-b5c4-25a22cb787ae"). InnerVolumeSpecName "kube-api-access-m4grg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:41:44.124687 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.124643    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-token" (OuterVolumeSpecName: "istio-token") pod "71c2f787-a899-4312-b5c4-25a22cb787ae" (UID: "71c2f787-a899-4312-b5c4-25a22cb787ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:41:44.124763 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.124742    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "71c2f787-a899-4312-b5c4-25a22cb787ae" (UID: "71c2f787-a899-4312-b5c4-25a22cb787ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:41:44.124839 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.124819    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "71c2f787-a899-4312-b5c4-25a22cb787ae" (UID: "71c2f787-a899-4312-b5c4-25a22cb787ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:44.209770 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.209732    2552 generic.go:356] "Generic (PLEG): container finished" podID="71c2f787-a899-4312-b5c4-25a22cb787ae" containerID="d0ce2e14922f1e3880a31e13ca1355f29566969b48d56b697ce108ff2cbe8454" exitCode=0
Sep 10 17:41:44.209970 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.209783    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"
Sep 10 17:41:44.209970 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.209815    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb" event={"ID":"71c2f787-a899-4312-b5c4-25a22cb787ae","Type":"ContainerDied","Data":"d0ce2e14922f1e3880a31e13ca1355f29566969b48d56b697ce108ff2cbe8454"}
Sep 10 17:41:44.209970 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.209910    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb" event={"ID":"71c2f787-a899-4312-b5c4-25a22cb787ae","Type":"ContainerDied","Data":"9eb61045a9f23d70e4f43583536c38b2d7d2ad9d41890ee2f443f64e79f74071"}
Sep 10 17:41:44.209970 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.209940    2552 scope.go:117] "RemoveContainer" containerID="d0ce2e14922f1e3880a31e13ca1355f29566969b48d56b697ce108ff2cbe8454"
Sep 10 17:41:44.219414 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.219389    2552 scope.go:117] "RemoveContainer" containerID="b12725a5050fc49da165e716b810f34cfbfb898934c206378492e8c0719e4523"
Sep 10 17:41:44.224049 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.224016    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-podinfo\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:44.224049 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.224043    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-workload-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:44.224232 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.224056    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-token\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:44.224232 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.224067    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-data\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:44.224232 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.224075    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c2f787-a899-4312-b5c4-25a22cb787ae-istio-envoy\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:44.224232 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.224083    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4grg\" (UniqueName: \"kubernetes.io/projected/71c2f787-a899-4312-b5c4-25a22cb787ae-kube-api-access-m4grg\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:41:44.228387 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.228366    2552 scope.go:117] "RemoveContainer" containerID="d0ce2e14922f1e3880a31e13ca1355f29566969b48d56b697ce108ff2cbe8454"
Sep 10 17:41:44.228739 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:41:44.228719    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ce2e14922f1e3880a31e13ca1355f29566969b48d56b697ce108ff2cbe8454\": container with ID starting with d0ce2e14922f1e3880a31e13ca1355f29566969b48d56b697ce108ff2cbe8454 not found: ID does not exist" containerID="d0ce2e14922f1e3880a31e13ca1355f29566969b48d56b697ce108ff2cbe8454"
Sep 10 17:41:44.228820 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.228752    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ce2e14922f1e3880a31e13ca1355f29566969b48d56b697ce108ff2cbe8454"} err="failed to get container status \"d0ce2e14922f1e3880a31e13ca1355f29566969b48d56b697ce108ff2cbe8454\": rpc error: code = NotFound desc = could not find container \"d0ce2e14922f1e3880a31e13ca1355f29566969b48d56b697ce108ff2cbe8454\": container with ID starting with d0ce2e14922f1e3880a31e13ca1355f29566969b48d56b697ce108ff2cbe8454 not found: ID does not exist"
Sep 10 17:41:44.228820 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.228781    2552 scope.go:117] "RemoveContainer" containerID="b12725a5050fc49da165e716b810f34cfbfb898934c206378492e8c0719e4523"
Sep 10 17:41:44.229064 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:41:44.229037    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b12725a5050fc49da165e716b810f34cfbfb898934c206378492e8c0719e4523\": container with ID starting with b12725a5050fc49da165e716b810f34cfbfb898934c206378492e8c0719e4523 not found: ID does not exist" containerID="b12725a5050fc49da165e716b810f34cfbfb898934c206378492e8c0719e4523"
Sep 10 17:41:44.229105 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.229071    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b12725a5050fc49da165e716b810f34cfbfb898934c206378492e8c0719e4523"} err="failed to get container status \"b12725a5050fc49da165e716b810f34cfbfb898934c206378492e8c0719e4523\": rpc error: code = NotFound desc = could not find container \"b12725a5050fc49da165e716b810f34cfbfb898934c206378492e8c0719e4523\": container with ID starting with b12725a5050fc49da165e716b810f34cfbfb898934c206378492e8c0719e4523 not found: ID does not exist"
Sep 10 17:41:44.235992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.235965    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"]
Sep 10 17:41:44.240113 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.240087    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2f1093-openshift-default-858454c7cd-hg2kb"]
Sep 10 17:41:44.804380 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:44.804348    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71c2f787-a899-4312-b5c4-25a22cb787ae" path="/var/lib/kubelet/pods/71c2f787-a899-4312-b5c4-25a22cb787ae/volumes"
Sep 10 17:41:47.169675 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:41:47.169631    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:42:06.325177 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325143    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"]
Sep 10 17:42:06.325703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325686    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c2f787-a899-4312-b5c4-25a22cb787ae" containerName="istio-proxy"
Sep 10 17:42:06.325754 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325705    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c2f787-a899-4312-b5c4-25a22cb787ae" containerName="istio-proxy"
Sep 10 17:42:06.325754 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325725    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c2f787-a899-4312-b5c4-25a22cb787ae" containerName="istio-proxy"
Sep 10 17:42:06.325754 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325733    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c2f787-a899-4312-b5c4-25a22cb787ae" containerName="istio-proxy"
Sep 10 17:42:06.325754 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325745    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" containerName="istio-proxy"
Sep 10 17:42:06.325754 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325753    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" containerName="istio-proxy"
Sep 10 17:42:06.325909 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325769    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67945228-9082-4a56-921c-516aec05cfcf" containerName="main"
Sep 10 17:42:06.325909 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325778    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="67945228-9082-4a56-921c-516aec05cfcf" containerName="main"
Sep 10 17:42:06.325909 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325793    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" containerName="istio-proxy"
Sep 10 17:42:06.325909 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325801    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" containerName="istio-proxy"
Sep 10 17:42:06.325909 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325816    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d772bb2-787c-439a-8e3c-c082df619445" containerName="main"
Sep 10 17:42:06.325909 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325825    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d772bb2-787c-439a-8e3c-c082df619445" containerName="main"
Sep 10 17:42:06.325909 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325908    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" containerName="istio-proxy"
Sep 10 17:42:06.326119 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325919    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="67945228-9082-4a56-921c-516aec05cfcf" containerName="main"
Sep 10 17:42:06.326119 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325928    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c2f787-a899-4312-b5c4-25a22cb787ae" containerName="istio-proxy"
Sep 10 17:42:06.326119 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325938    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c2f787-a899-4312-b5c4-25a22cb787ae" containerName="istio-proxy"
Sep 10 17:42:06.326119 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325948    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d772bb2-787c-439a-8e3c-c082df619445" containerName="main"
Sep 10 17:42:06.326119 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.325960    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="289be72d-8c4c-4188-b86e-ce5dfd88ae92" containerName="istio-proxy"
Sep 10 17:42:06.329612 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.329585    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.332877 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.332851    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 17:42:06.333019 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.332875    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-762738-openshift-default-dockercfg-rjf79\""
Sep 10 17:42:06.341758 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.341732    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"]
Sep 10 17:42:06.424747 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.424702    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493420a1-4c9f-46dd-9819-9ad48049d271-istio-podinfo\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.424954 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.424754    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-istio-data\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.424954 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.424799    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-workload-socket\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.424954 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.424836    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493420a1-4c9f-46dd-9819-9ad48049d271-istiod-ca-cert\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.424954 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.424921    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-workload-certs\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.425184 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.424955    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-istio-envoy\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.425184 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.424987    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc5dz\" (UniqueName: \"kubernetes.io/projected/493420a1-4c9f-46dd-9819-9ad48049d271-kube-api-access-sc5dz\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.425184 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.425072    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493420a1-4c9f-46dd-9819-9ad48049d271-istio-token\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.425184 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.425134    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-credential-socket\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.526339 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.526299    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-workload-socket\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.526339 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.526345    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493420a1-4c9f-46dd-9819-9ad48049d271-istiod-ca-cert\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.526635 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.526388    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-workload-certs\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.526635 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.526409    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-istio-envoy\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.526635 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.526434    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc5dz\" (UniqueName: \"kubernetes.io/projected/493420a1-4c9f-46dd-9819-9ad48049d271-kube-api-access-sc5dz\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.526635 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.526461    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493420a1-4c9f-46dd-9819-9ad48049d271-istio-token\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.526635 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.526538    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-credential-socket\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.526635 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.526593    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493420a1-4c9f-46dd-9819-9ad48049d271-istio-podinfo\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.526959 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.526637    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-istio-data\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.526959 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.526842    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-workload-socket\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.526959 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.526916    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-credential-socket\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.527104 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.526964    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-istio-data\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.527167 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.527106    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-workload-certs\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.527250 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.527229    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493420a1-4c9f-46dd-9819-9ad48049d271-istiod-ca-cert\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.528840 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.528818    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-istio-envoy\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.529092 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.529071    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493420a1-4c9f-46dd-9819-9ad48049d271-istio-podinfo\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.534979 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.534956    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493420a1-4c9f-46dd-9819-9ad48049d271-istio-token\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.535164 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.535145    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc5dz\" (UniqueName: \"kubernetes.io/projected/493420a1-4c9f-46dd-9819-9ad48049d271-kube-api-access-sc5dz\") pod \"e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") " pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.642435 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.642397    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:06.926618 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.926426    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"]
Sep 10 17:42:06.928924 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:42:06.928895    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493420a1_4c9f_46dd_9819_9ad48049d271.slice/crio-0b0a8a768705327e184aeb007515d929cbc48aba39b419fc474f8b35cc805b69 WatchSource:0}: Error finding container 0b0a8a768705327e184aeb007515d929cbc48aba39b419fc474f8b35cc805b69: Status 404 returned error can't find the container with id 0b0a8a768705327e184aeb007515d929cbc48aba39b419fc474f8b35cc805b69
Sep 10 17:42:06.931187 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.931147    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:42:06.931281 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.931218    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:42:06.931281 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:06.931245    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:42:07.334095 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:07.334054    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw" event={"ID":"493420a1-4c9f-46dd-9819-9ad48049d271","Type":"ContainerStarted","Data":"929a03862aec8a765099779195bc7fbc3a336fa405d3cd8429c0900d2a2f385c"}
Sep 10 17:42:07.334463 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:07.334158    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw" event={"ID":"493420a1-4c9f-46dd-9819-9ad48049d271","Type":"ContainerStarted","Data":"0b0a8a768705327e184aeb007515d929cbc48aba39b419fc474f8b35cc805b69"}
Sep 10 17:42:07.354842 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:07.354783    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw" podStartSLOduration=1.354762898 podStartE2EDuration="1.354762898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:07.354389286 +0000 UTC m=+1099.112916357" watchObservedRunningTime="2026-09-10 17:42:07.354762898 +0000 UTC m=+1099.113289970"
Sep 10 17:42:07.642704 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:07.642611    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:07.647742 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:07.647715    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:08.261251 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.261218    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"]
Sep 10 17:42:08.261881 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.261866    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c4b9288-cfb9-4afc-aa75-f85d7f61c66b" containerName="istio-proxy"
Sep 10 17:42:08.264367 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.264350    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.267306 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.267270    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-4f14bf-openshift-default-dockercfg-vrcm2\""
Sep 10 17:42:08.281035 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.280998    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"]
Sep 10 17:42:08.340179 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.340144    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:08.341193 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.341172    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:08.345391 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.345365    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-credential-socket\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.345530 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.345407    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b52144b-be1b-47fd-a056-46f645501009-istio-podinfo\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.345530 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.345439    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b52144b-be1b-47fd-a056-46f645501009-istio-token\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.345530 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.345463    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv2j8\" (UniqueName: \"kubernetes.io/projected/1b52144b-be1b-47fd-a056-46f645501009-kube-api-access-sv2j8\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.345708 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.345592    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-istio-envoy\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.345708 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.345618    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-istio-data\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.345708 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.345656    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b52144b-be1b-47fd-a056-46f645501009-istiod-ca-cert\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.345829 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.345705    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-workload-socket\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.345829 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.345730    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-workload-certs\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.447276 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.447237    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-credential-socket\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.447276 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.447278    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b52144b-be1b-47fd-a056-46f645501009-istio-podinfo\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.447549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.447299    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b52144b-be1b-47fd-a056-46f645501009-istio-token\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.447549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.447317    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv2j8\" (UniqueName: \"kubernetes.io/projected/1b52144b-be1b-47fd-a056-46f645501009-kube-api-access-sv2j8\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.447549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.447416    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-istio-envoy\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.447549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.447441    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-istio-data\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.447549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.447475    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b52144b-be1b-47fd-a056-46f645501009-istiod-ca-cert\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.447549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.447524    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-workload-socket\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.447549 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.447549    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-workload-certs\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.447928 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.447736    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-credential-socket\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.448052 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.447994    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-workload-certs\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.448150 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.448046    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-workload-socket\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.448259 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.448236    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-istio-data\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.448589 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.448566    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b52144b-be1b-47fd-a056-46f645501009-istiod-ca-cert\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.450214 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.450194    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-istio-envoy\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.450452 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.450433    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b52144b-be1b-47fd-a056-46f645501009-istio-podinfo\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.455645 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.455621    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b52144b-be1b-47fd-a056-46f645501009-istio-token\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.456028 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.456006    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv2j8\" (UniqueName: \"kubernetes.io/projected/1b52144b-be1b-47fd-a056-46f645501009-kube-api-access-sv2j8\") pod \"e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") " pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.576975 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.576874    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:08.851909 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.851886    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"]
Sep 10 17:42:08.854418 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:42:08.854390    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b52144b_be1b_47fd_a056_46f645501009.slice/crio-63803e6e5a107e637bd9b3c170b07b7a6a76c682d35d6c9e4a87d1f24c479744 WatchSource:0}: Error finding container 63803e6e5a107e637bd9b3c170b07b7a6a76c682d35d6c9e4a87d1f24c479744: Status 404 returned error can't find the container with id 63803e6e5a107e637bd9b3c170b07b7a6a76c682d35d6c9e4a87d1f24c479744
Sep 10 17:42:08.857258 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.857222    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:42:08.857359 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.857297    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:42:08.857359 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:08.857327    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:42:09.346870 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:09.346832    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs" event={"ID":"1b52144b-be1b-47fd-a056-46f645501009","Type":"ContainerStarted","Data":"6f35d9c9dbf3b73b8e14582535a1683c588d938bf60218637d73aef89bfa45f3"}
Sep 10 17:42:09.347249 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:09.346897    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs" event={"ID":"1b52144b-be1b-47fd-a056-46f645501009","Type":"ContainerStarted","Data":"63803e6e5a107e637bd9b3c170b07b7a6a76c682d35d6c9e4a87d1f24c479744"}
Sep 10 17:42:09.367236 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:09.367186    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs" podStartSLOduration=1.367169946 podStartE2EDuration="1.367169946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:09.366652331 +0000 UTC m=+1101.125179401" watchObservedRunningTime="2026-09-10 17:42:09.367169946 +0000 UTC m=+1101.125697016"
Sep 10 17:42:09.577056 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:09.577011    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:09.581814 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:09.581783    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:10.356485 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:10.356438    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:10.357738 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:10.357712    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:16.504686 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:16.504642    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:42:26.398670 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:26.398621    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 17:42:26.399109 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:26.398712    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs" podUID="1b52144b-be1b-47fd-a056-46f645501009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 17:42:27.447853 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:27.447816    2552 generic.go:356] "Generic (PLEG): container finished" podID="493420a1-4c9f-46dd-9819-9ad48049d271" containerID="929a03862aec8a765099779195bc7fbc3a336fa405d3cd8429c0900d2a2f385c" exitCode=0
Sep 10 17:42:27.448357 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:27.447889    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw" event={"ID":"493420a1-4c9f-46dd-9819-9ad48049d271","Type":"ContainerDied","Data":"929a03862aec8a765099779195bc7fbc3a336fa405d3cd8429c0900d2a2f385c"}
Sep 10 17:42:27.448414 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:27.448369    2552 scope.go:117] "RemoveContainer" containerID="929a03862aec8a765099779195bc7fbc3a336fa405d3cd8429c0900d2a2f385c"
Sep 10 17:42:27.449586 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:27.449563    2552 generic.go:356] "Generic (PLEG): container finished" podID="1b52144b-be1b-47fd-a056-46f645501009" containerID="6f35d9c9dbf3b73b8e14582535a1683c588d938bf60218637d73aef89bfa45f3" exitCode=0
Sep 10 17:42:27.449664 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:27.449621    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs" event={"ID":"1b52144b-be1b-47fd-a056-46f645501009","Type":"ContainerDied","Data":"6f35d9c9dbf3b73b8e14582535a1683c588d938bf60218637d73aef89bfa45f3"}
Sep 10 17:42:27.449703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:27.449620    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:42:27.449767 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:27.449751    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:42:27.449809 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:27.449791    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:42:27.449983 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:27.449971    2552 scope.go:117] "RemoveContainer" containerID="6f35d9c9dbf3b73b8e14582535a1683c588d938bf60218637d73aef89bfa45f3"
Sep 10 17:42:27.450976 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:27.450949    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:42:27.451045 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:27.451028    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:42:27.451104 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:27.451075    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:42:28.462339 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:28.462304    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw" event={"ID":"493420a1-4c9f-46dd-9819-9ad48049d271","Type":"ContainerStarted","Data":"103cfb12d7dfa7791d124f3c3f3d51fdafbefb9cb8e06afc9513362681966889"}
Sep 10 17:42:28.468746 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:28.468703    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs" event={"ID":"1b52144b-be1b-47fd-a056-46f645501009","Type":"ContainerStarted","Data":"ebe5992f1eabbe6ea07988ebf80c98493418f9bfd025d1aee1dce8675324544b"}
Sep 10 17:42:28.577675 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:28.577631    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:28.578224 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:28.578188    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 10 17:42:28.581118 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:28.581082    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs" podUID="1b52144b-be1b-47fd-a056-46f645501009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 10 17:42:28.643670 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:28.643589    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:28.644032 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:28.644006    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 10 17:42:28.644259 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:28.644222    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw" podUID="493420a1-4c9f-46dd-9819-9ad48049d271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 10 17:42:29.578057 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:29.578004    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 10 17:42:29.578559 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:29.578081    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs" podUID="1b52144b-be1b-47fd-a056-46f645501009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 10 17:42:29.643450 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:29.643413    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 10 17:42:29.643685 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:29.643480    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw" podUID="493420a1-4c9f-46dd-9819-9ad48049d271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 10 17:42:30.577473 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:30.577423    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 10 17:42:30.577705 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:30.577516    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs" podUID="1b52144b-be1b-47fd-a056-46f645501009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 10 17:42:30.642969 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:30.642931    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 10 17:42:30.643429 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:30.642997    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw" podUID="493420a1-4c9f-46dd-9819-9ad48049d271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 10 17:42:31.577790 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:31.577742    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 10 17:42:31.578026 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:31.577817    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs" podUID="1b52144b-be1b-47fd-a056-46f645501009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 10 17:42:31.643047 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:31.643004    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 10 17:42:31.643434 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:31.643061    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw" podUID="493420a1-4c9f-46dd-9819-9ad48049d271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 10 17:42:32.578182 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:32.578141    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 10 17:42:32.578465 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:32.578439    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs" podUID="1b52144b-be1b-47fd-a056-46f645501009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 10 17:42:32.643227 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:32.643194    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 10 17:42:32.643772 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:32.643740    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw" podUID="493420a1-4c9f-46dd-9819-9ad48049d271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 10 17:42:33.577441 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:33.577405    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 10 17:42:33.577658 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:33.577466    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs" podUID="1b52144b-be1b-47fd-a056-46f645501009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 10 17:42:33.827514 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:33.827414    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:33.828208 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:33.828181    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:33.831445 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:33.831424    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:42:34.581417 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:34.581386    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:34.581799 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:34.581780    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:34.582468 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:34.582446    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:42:34.960452 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:34.960410    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-9e4de5"]
Sep 10 17:42:34.963171 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:34.963153    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-9e4de5"
Sep 10 17:42:34.966076 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:34.966052    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p5m84\""
Sep 10 17:42:34.968354 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:34.968327    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-9e4de5"]
Sep 10 17:42:35.047845 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:35.047802    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-799hg\" (UniqueName: \"kubernetes.io/projected/43d587d3-567e-4f0e-b37d-dcf5cdb6c639-kube-api-access-799hg\") pod \"test-curl-7761-9e4de5\" (UID: \"43d587d3-567e-4f0e-b37d-dcf5cdb6c639\") " pod="opendatahub/test-curl-7761-9e4de5"
Sep 10 17:42:35.148760 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:35.148719    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-799hg\" (UniqueName: \"kubernetes.io/projected/43d587d3-567e-4f0e-b37d-dcf5cdb6c639-kube-api-access-799hg\") pod \"test-curl-7761-9e4de5\" (UID: \"43d587d3-567e-4f0e-b37d-dcf5cdb6c639\") " pod="opendatahub/test-curl-7761-9e4de5"
Sep 10 17:42:35.157706 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:35.157681    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-799hg\" (UniqueName: \"kubernetes.io/projected/43d587d3-567e-4f0e-b37d-dcf5cdb6c639-kube-api-access-799hg\") pod \"test-curl-7761-9e4de5\" (UID: \"43d587d3-567e-4f0e-b37d-dcf5cdb6c639\") " pod="opendatahub/test-curl-7761-9e4de5"
Sep 10 17:42:35.274131 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:35.274048    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-9e4de5"
Sep 10 17:42:35.548221 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:35.548193    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-9e4de5"]
Sep 10 17:42:35.550410 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:42:35.550379    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43d587d3_567e_4f0e_b37d_dcf5cdb6c639.slice/crio-b6357abc41896a2d17a69f06c402d8aee044663a13c1cd8b9ca95d9d820c2c12 WatchSource:0}: Error finding container b6357abc41896a2d17a69f06c402d8aee044663a13c1cd8b9ca95d9d820c2c12: Status 404 returned error can't find the container with id b6357abc41896a2d17a69f06c402d8aee044663a13c1cd8b9ca95d9d820c2c12
Sep 10 17:42:35.552243 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:35.552224    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:42:36.522698 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:36.522654    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-9e4de5" event={"ID":"43d587d3-567e-4f0e-b37d-dcf5cdb6c639","Type":"ContainerStarted","Data":"b6357abc41896a2d17a69f06c402d8aee044663a13c1cd8b9ca95d9d820c2c12"}
Sep 10 17:42:38.536093 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:38.536054    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-9e4de5" event={"ID":"43d587d3-567e-4f0e-b37d-dcf5cdb6c639","Type":"ContainerStarted","Data":"bc9e21e9ac8eb4f3fc751139a4b61955180c162f9e132059f17acea4372a9b7b"}
Sep 10 17:42:38.552260 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:38.552206    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7761-9e4de5" podStartSLOduration=2.610734579 podStartE2EDuration="4.552189171s" totalImagesPullingTime="1.941454592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:38.551250853 +0000 UTC m=+1130.309777924" watchObservedRunningTime="2026-09-10 17:42:38.552189171 +0000 UTC m=+1130.310716254"
Sep 10 17:42:39.168705 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:39.168667    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-9e4de5"]
Sep 10 17:42:39.173309 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:39.173276    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-9e4de5"]
Sep 10 17:42:39.286411 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:39.286328    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-22c1b2"]
Sep 10 17:42:39.286738 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:39.286724    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d587d3-567e-4f0e-b37d-dcf5cdb6c639" containerName="test-curl-7761-9e4de5"
Sep 10 17:42:39.286790 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:39.286739    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d587d3-567e-4f0e-b37d-dcf5cdb6c639" containerName="test-curl-7761-9e4de5"
Sep 10 17:42:39.286828 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:39.286819    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d587d3-567e-4f0e-b37d-dcf5cdb6c639" containerName="test-curl-7761-9e4de5"
Sep 10 17:42:39.289214 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:39.289195    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-22c1b2"
Sep 10 17:42:39.295141 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:39.295111    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-22c1b2"]
Sep 10 17:42:39.392560 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:39.392524    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlft7\" (UniqueName: \"kubernetes.io/projected/fffa5673-e74f-495d-bf2b-66dd1927971c-kube-api-access-dlft7\") pod \"test-curl-7761-22c1b2\" (UID: \"fffa5673-e74f-495d-bf2b-66dd1927971c\") " pod="opendatahub/test-curl-7761-22c1b2"
Sep 10 17:42:39.494167 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:39.494124    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlft7\" (UniqueName: \"kubernetes.io/projected/fffa5673-e74f-495d-bf2b-66dd1927971c-kube-api-access-dlft7\") pod \"test-curl-7761-22c1b2\" (UID: \"fffa5673-e74f-495d-bf2b-66dd1927971c\") " pod="opendatahub/test-curl-7761-22c1b2"
Sep 10 17:42:39.505308 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:39.505278    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlft7\" (UniqueName: \"kubernetes.io/projected/fffa5673-e74f-495d-bf2b-66dd1927971c-kube-api-access-dlft7\") pod \"test-curl-7761-22c1b2\" (UID: \"fffa5673-e74f-495d-bf2b-66dd1927971c\") " pod="opendatahub/test-curl-7761-22c1b2"
Sep 10 17:42:39.541892 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:39.541806    2552 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7761-9e4de5" container="test-curl-7761-9e4de5"
Sep 10 17:42:39.600588 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:39.600554    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-22c1b2"
Sep 10 17:42:39.872777 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:39.872451    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-22c1b2"]
Sep 10 17:42:39.875411 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:42:39.875383    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfffa5673_e74f_495d_bf2b_66dd1927971c.slice/crio-7d731558f04274740f9b44487c71a94baa0ac82b237aac00137637a98cf68334 WatchSource:0}: Error finding container 7d731558f04274740f9b44487c71a94baa0ac82b237aac00137637a98cf68334: Status 404 returned error can't find the container with id 7d731558f04274740f9b44487c71a94baa0ac82b237aac00137637a98cf68334
Sep 10 17:42:40.548223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:40.548149    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-22c1b2" event={"ID":"fffa5673-e74f-495d-bf2b-66dd1927971c","Type":"ContainerStarted","Data":"00b202f5dfd7abf34fb5b0d8619c30cc556bebcc87e10252014034f87eaa2b43"}
Sep 10 17:42:40.548721 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:40.548587    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-22c1b2" event={"ID":"fffa5673-e74f-495d-bf2b-66dd1927971c","Type":"ContainerStarted","Data":"7d731558f04274740f9b44487c71a94baa0ac82b237aac00137637a98cf68334"}
Sep 10 17:42:40.548721 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:40.548630    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7761-9e4de5" podUID="43d587d3-567e-4f0e-b37d-dcf5cdb6c639" containerName="test-curl-7761-9e4de5" containerID="cri-o://bc9e21e9ac8eb4f3fc751139a4b61955180c162f9e132059f17acea4372a9b7b" gracePeriod=2
Sep 10 17:42:40.553957 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:40.553917    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7761-9e4de5\" is forbidden: User \"system:node:ip-10-0-129-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-90.ec2.internal' and this object" podUID="43d587d3-567e-4f0e-b37d-dcf5cdb6c639" pod="opendatahub/test-curl-7761-9e4de5"
Sep 10 17:42:40.558709 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:40.558677    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7761-9e4de5\" is forbidden: User \"system:node:ip-10-0-129-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-90.ec2.internal' and this object" podUID="43d587d3-567e-4f0e-b37d-dcf5cdb6c639" pod="opendatahub/test-curl-7761-9e4de5"
Sep 10 17:42:40.587922 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:40.587868    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7761-22c1b2" podStartSLOduration=1.5878506479999999 podStartE2EDuration="1.587850648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:40.585948319 +0000 UTC m=+1132.344475388" watchObservedRunningTime="2026-09-10 17:42:40.587850648 +0000 UTC m=+1132.346377718"
Sep 10 17:42:41.194378 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:41.194329    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-22c1b2"]
Sep 10 17:42:41.197456 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:41.197419    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-22c1b2"]
Sep 10 17:42:41.310846 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:41.310767    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-0268f7"]
Sep 10 17:42:41.311170 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:41.311156    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffa5673-e74f-495d-bf2b-66dd1927971c" containerName="test-curl-7761-22c1b2"
Sep 10 17:42:41.311226 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:41.311171    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffa5673-e74f-495d-bf2b-66dd1927971c" containerName="test-curl-7761-22c1b2"
Sep 10 17:42:41.311261 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:41.311231    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffa5673-e74f-495d-bf2b-66dd1927971c" containerName="test-curl-7761-22c1b2"
Sep 10 17:42:41.313427 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:41.313410    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-0268f7"
Sep 10 17:42:41.319485 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:41.319455    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-0268f7"]
Sep 10 17:42:41.413589 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:41.413545    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hltk7\" (UniqueName: \"kubernetes.io/projected/4de4d08d-bfe8-4b48-bb03-e6b90c863728-kube-api-access-hltk7\") pod \"test-curl-7761-0268f7\" (UID: \"4de4d08d-bfe8-4b48-bb03-e6b90c863728\") " pod="opendatahub/test-curl-7761-0268f7"
Sep 10 17:42:41.514437 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:41.514391    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hltk7\" (UniqueName: \"kubernetes.io/projected/4de4d08d-bfe8-4b48-bb03-e6b90c863728-kube-api-access-hltk7\") pod \"test-curl-7761-0268f7\" (UID: \"4de4d08d-bfe8-4b48-bb03-e6b90c863728\") " pod="opendatahub/test-curl-7761-0268f7"
Sep 10 17:42:41.523152 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:41.523124    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hltk7\" (UniqueName: \"kubernetes.io/projected/4de4d08d-bfe8-4b48-bb03-e6b90c863728-kube-api-access-hltk7\") pod \"test-curl-7761-0268f7\" (UID: \"4de4d08d-bfe8-4b48-bb03-e6b90c863728\") " pod="opendatahub/test-curl-7761-0268f7"
Sep 10 17:42:41.553649 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:41.553615    2552 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7761-22c1b2" container="test-curl-7761-22c1b2"
Sep 10 17:42:41.625206 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:41.625175    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-0268f7"
Sep 10 17:42:41.900657 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:41.900626    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-0268f7"]
Sep 10 17:42:41.902082 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:42:41.902047    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4de4d08d_bfe8_4b48_bb03_e6b90c863728.slice/crio-ea666306076021259ef87a5faf5408bf32904f6cfa910ab3d97614265071c5e2 WatchSource:0}: Error finding container ea666306076021259ef87a5faf5408bf32904f6cfa910ab3d97614265071c5e2: Status 404 returned error can't find the container with id ea666306076021259ef87a5faf5408bf32904f6cfa910ab3d97614265071c5e2
Sep 10 17:42:42.560194 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.560160    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-0268f7" event={"ID":"4de4d08d-bfe8-4b48-bb03-e6b90c863728","Type":"ContainerStarted","Data":"23d5ecd5ef9a181159dea481be6d79bab78eb57f34a4673dd08aca6eb2afb1e3"}
Sep 10 17:42:42.560638 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.560203    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-0268f7" event={"ID":"4de4d08d-bfe8-4b48-bb03-e6b90c863728","Type":"ContainerStarted","Data":"ea666306076021259ef87a5faf5408bf32904f6cfa910ab3d97614265071c5e2"}
Sep 10 17:42:42.560638 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.560316    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7761-22c1b2" podUID="fffa5673-e74f-495d-bf2b-66dd1927971c" containerName="test-curl-7761-22c1b2" containerID="cri-o://00b202f5dfd7abf34fb5b0d8619c30cc556bebcc87e10252014034f87eaa2b43" gracePeriod=2
Sep 10 17:42:42.577454 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.577396    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7761-0268f7" podStartSLOduration=1.5773779829999999 podStartE2EDuration="1.577377983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:42.575508023 +0000 UTC m=+1134.334035117" watchObservedRunningTime="2026-09-10 17:42:42.577377983 +0000 UTC m=+1134.335905053"
Sep 10 17:42:42.578048 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:42.578025    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7761-22c1b2\" is forbidden: User \"system:node:ip-10-0-129-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-90.ec2.internal' and this object" podUID="fffa5673-e74f-495d-bf2b-66dd1927971c" pod="opendatahub/test-curl-7761-22c1b2"
Sep 10 17:42:42.739121 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.739094    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-9e4de5"
Sep 10 17:42:42.741468 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:42.741440    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7761-22c1b2\" is forbidden: User \"system:node:ip-10-0-129-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-90.ec2.internal' and this object" podUID="fffa5673-e74f-495d-bf2b-66dd1927971c" pod="opendatahub/test-curl-7761-22c1b2"
Sep 10 17:42:42.805174 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.805147    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"]
Sep 10 17:42:42.807459 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.807435    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:42.810810 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.810739    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-058069\"/\"default-dockercfg-fz8ts\""
Sep 10 17:42:42.810810 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.810742    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-058069\"/\"auth-test-model-058069-kserve-self-signed-certs\""
Sep 10 17:42:42.811950 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.811900    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-058069\"/\"openshift-service-ca.crt\""
Sep 10 17:42:42.811950 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.811918    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-058069\"/\"kube-root-ca.crt\""
Sep 10 17:42:42.815429 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.815402    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"]
Sep 10 17:42:42.826333 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.826304    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-799hg\" (UniqueName: \"kubernetes.io/projected/43d587d3-567e-4f0e-b37d-dcf5cdb6c639-kube-api-access-799hg\") pod \"43d587d3-567e-4f0e-b37d-dcf5cdb6c639\" (UID: \"43d587d3-567e-4f0e-b37d-dcf5cdb6c639\") "
Sep 10 17:42:42.828477 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.828445    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d587d3-567e-4f0e-b37d-dcf5cdb6c639-kube-api-access-799hg" (OuterVolumeSpecName: "kube-api-access-799hg") pod "43d587d3-567e-4f0e-b37d-dcf5cdb6c639" (UID: "43d587d3-567e-4f0e-b37d-dcf5cdb6c639"). InnerVolumeSpecName "kube-api-access-799hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:42:42.927299 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.927259    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-home\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:42.927299 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.927300    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb5e4c2-c980-49f7-bd61-23d9994f6491-tls-certs\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:42.927602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.927331    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-tmp-dir\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:42.927602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.927428    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxpbn\" (UniqueName: \"kubernetes.io/projected/bfb5e4c2-c980-49f7-bd61-23d9994f6491-kube-api-access-mxpbn\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:42.927602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.927462    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-dshm\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:42.927602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.927570    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-model-cache\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:42.927753 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:42.927654    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-799hg\" (UniqueName: \"kubernetes.io/projected/43d587d3-567e-4f0e-b37d-dcf5cdb6c639-kube-api-access-799hg\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:42:43.028943 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.028908    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-model-cache\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:43.029132 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.028989    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-home\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:43.029132 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.029006    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb5e4c2-c980-49f7-bd61-23d9994f6491-tls-certs\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:43.029132 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.029048    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-tmp-dir\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:43.029132 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.029093    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxpbn\" (UniqueName: \"kubernetes.io/projected/bfb5e4c2-c980-49f7-bd61-23d9994f6491-kube-api-access-mxpbn\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:43.029132 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.029120    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-dshm\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:43.029428 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.029405    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-model-cache\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:43.029678 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.029648    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-tmp-dir\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:43.029783 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.029648    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-home\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:43.031732 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.031699    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-dshm\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:43.031860 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.031805    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb5e4c2-c980-49f7-bd61-23d9994f6491-tls-certs\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:43.039580 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.039539    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxpbn\" (UniqueName: \"kubernetes.io/projected/bfb5e4c2-c980-49f7-bd61-23d9994f6491-kube-api-access-mxpbn\") pod \"auth-test-model-058069-kserve-8d7544c5-556z2\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") " pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:43.118007 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.117966    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:43.163187 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.163144    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-0268f7"]
Sep 10 17:42:43.168299 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.168178    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-0268f7"]
Sep 10 17:42:43.276641 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.276609    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-e15c04"]
Sep 10 17:42:43.276998 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.276985    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4de4d08d-bfe8-4b48-bb03-e6b90c863728" containerName="test-curl-7761-0268f7"
Sep 10 17:42:43.277069 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.277000    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4de4d08d-bfe8-4b48-bb03-e6b90c863728" containerName="test-curl-7761-0268f7"
Sep 10 17:42:43.277111 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.277081    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4de4d08d-bfe8-4b48-bb03-e6b90c863728" containerName="test-curl-7761-0268f7"
Sep 10 17:42:43.279678 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.279661    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-e15c04"
Sep 10 17:42:43.283863 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.283834    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-e15c04"]
Sep 10 17:42:43.332418 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.332380    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4j8l\" (UniqueName: \"kubernetes.io/projected/e5cebdb0-6bc3-495a-862a-1beac529f251-kube-api-access-g4j8l\") pod \"test-curl-7761-e15c04\" (UID: \"e5cebdb0-6bc3-495a-862a-1beac529f251\") " pod="opendatahub/test-curl-7761-e15c04"
Sep 10 17:42:43.389698 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.389670    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"]
Sep 10 17:42:43.391223 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:42:43.391196    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfb5e4c2_c980_49f7_bd61_23d9994f6491.slice/crio-409b228e4f571d357631102051b6eaf26fcb5ed9273dfd17f556ac8f8fa77ff5 WatchSource:0}: Error finding container 409b228e4f571d357631102051b6eaf26fcb5ed9273dfd17f556ac8f8fa77ff5: Status 404 returned error can't find the container with id 409b228e4f571d357631102051b6eaf26fcb5ed9273dfd17f556ac8f8fa77ff5
Sep 10 17:42:43.433159 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.433132    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4j8l\" (UniqueName: \"kubernetes.io/projected/e5cebdb0-6bc3-495a-862a-1beac529f251-kube-api-access-g4j8l\") pod \"test-curl-7761-e15c04\" (UID: \"e5cebdb0-6bc3-495a-862a-1beac529f251\") " pod="opendatahub/test-curl-7761-e15c04"
Sep 10 17:42:43.441554 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.441532    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4j8l\" (UniqueName: \"kubernetes.io/projected/e5cebdb0-6bc3-495a-862a-1beac529f251-kube-api-access-g4j8l\") pod \"test-curl-7761-e15c04\" (UID: \"e5cebdb0-6bc3-495a-862a-1beac529f251\") " pod="opendatahub/test-curl-7761-e15c04"
Sep 10 17:42:43.567995 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.567955    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2" event={"ID":"bfb5e4c2-c980-49f7-bd61-23d9994f6491","Type":"ContainerStarted","Data":"ebea64b73d662edf9bc7f0d35ede8d39501611b3a1dda4a56679b23f56750670"}
Sep 10 17:42:43.568442 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.568008    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2" event={"ID":"bfb5e4c2-c980-49f7-bd61-23d9994f6491","Type":"ContainerStarted","Data":"409b228e4f571d357631102051b6eaf26fcb5ed9273dfd17f556ac8f8fa77ff5"}
Sep 10 17:42:43.569242 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.569214    2552 generic.go:356] "Generic (PLEG): container finished" podID="43d587d3-567e-4f0e-b37d-dcf5cdb6c639" containerID="bc9e21e9ac8eb4f3fc751139a4b61955180c162f9e132059f17acea4372a9b7b" exitCode=137
Sep 10 17:42:43.569384 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.569261    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-9e4de5"
Sep 10 17:42:43.569384 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.569317    2552 scope.go:117] "RemoveContainer" containerID="bc9e21e9ac8eb4f3fc751139a4b61955180c162f9e132059f17acea4372a9b7b"
Sep 10 17:42:43.569485 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:43.569464    2552 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7761-0268f7" container="test-curl-7761-0268f7"
Sep 10 17:42:43.581592 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.581477    2552 scope.go:117] "RemoveContainer" containerID="bc9e21e9ac8eb4f3fc751139a4b61955180c162f9e132059f17acea4372a9b7b"
Sep 10 17:42:43.582944 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:43.582918    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc9e21e9ac8eb4f3fc751139a4b61955180c162f9e132059f17acea4372a9b7b\": container with ID starting with bc9e21e9ac8eb4f3fc751139a4b61955180c162f9e132059f17acea4372a9b7b not found: ID does not exist" containerID="bc9e21e9ac8eb4f3fc751139a4b61955180c162f9e132059f17acea4372a9b7b"
Sep 10 17:42:43.583073 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.582956    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9e21e9ac8eb4f3fc751139a4b61955180c162f9e132059f17acea4372a9b7b"} err="failed to get container status \"bc9e21e9ac8eb4f3fc751139a4b61955180c162f9e132059f17acea4372a9b7b\": rpc error: code = NotFound desc = could not find container \"bc9e21e9ac8eb4f3fc751139a4b61955180c162f9e132059f17acea4372a9b7b\": container with ID starting with bc9e21e9ac8eb4f3fc751139a4b61955180c162f9e132059f17acea4372a9b7b not found: ID does not exist"
Sep 10 17:42:43.589022 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.588980    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2" podStartSLOduration=1.5889673640000002 podStartE2EDuration="1.588967364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:43.588074511 +0000 UTC m=+1135.346601580" watchObservedRunningTime="2026-09-10 17:42:43.588967364 +0000 UTC m=+1135.347494434"
Sep 10 17:42:43.590723 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.590700    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-e15c04"
Sep 10 17:42:43.862823 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:43.862796    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-e15c04"]
Sep 10 17:42:43.865065 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:42:43.865018    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5cebdb0_6bc3_495a_862a_1beac529f251.slice/crio-2c91cf2514c61d955c6271f9b05a3593fa5e9381bc4aeab03ed6ed6db508e23e WatchSource:0}: Error finding container 2c91cf2514c61d955c6271f9b05a3593fa5e9381bc4aeab03ed6ed6db508e23e: Status 404 returned error can't find the container with id 2c91cf2514c61d955c6271f9b05a3593fa5e9381bc4aeab03ed6ed6db508e23e
Sep 10 17:42:44.578204 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:44.578170    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-e15c04" event={"ID":"e5cebdb0-6bc3-495a-862a-1beac529f251","Type":"ContainerStarted","Data":"0a44f633667c5b4416559fed03ebc8acff6426eb5bc25465fe9a922e52977741"}
Sep 10 17:42:44.578531 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:44.578221    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-e15c04" event={"ID":"e5cebdb0-6bc3-495a-862a-1beac529f251","Type":"ContainerStarted","Data":"2c91cf2514c61d955c6271f9b05a3593fa5e9381bc4aeab03ed6ed6db508e23e"}
Sep 10 17:42:44.578531 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:44.578454    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7761-0268f7" podUID="4de4d08d-bfe8-4b48-bb03-e6b90c863728" containerName="test-curl-7761-0268f7" containerID="cri-o://23d5ecd5ef9a181159dea481be6d79bab78eb57f34a4673dd08aca6eb2afb1e3" gracePeriod=2
Sep 10 17:42:44.594761 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:44.594716    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7761-e15c04" podStartSLOduration=1.594699519 podStartE2EDuration="1.594699519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:44.593718442 +0000 UTC m=+1136.352245511" watchObservedRunningTime="2026-09-10 17:42:44.594699519 +0000 UTC m=+1136.353226588"
Sep 10 17:42:44.596555 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:44.596524    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7761-0268f7\" is forbidden: User \"system:node:ip-10-0-129-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-90.ec2.internal' and this object" podUID="4de4d08d-bfe8-4b48-bb03-e6b90c863728" pod="opendatahub/test-curl-7761-0268f7"
Sep 10 17:42:44.733975 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:44.733948    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-22c1b2"
Sep 10 17:42:44.736614 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:44.736585    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7761-0268f7\" is forbidden: User \"system:node:ip-10-0-129-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-90.ec2.internal' and this object" podUID="4de4d08d-bfe8-4b48-bb03-e6b90c863728" pod="opendatahub/test-curl-7761-0268f7"
Sep 10 17:42:44.804182 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:44.804147    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43d587d3-567e-4f0e-b37d-dcf5cdb6c639" path="/var/lib/kubelet/pods/43d587d3-567e-4f0e-b37d-dcf5cdb6c639/volumes"
Sep 10 17:42:44.847584 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:44.847554    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlft7\" (UniqueName: \"kubernetes.io/projected/fffa5673-e74f-495d-bf2b-66dd1927971c-kube-api-access-dlft7\") pod \"fffa5673-e74f-495d-bf2b-66dd1927971c\" (UID: \"fffa5673-e74f-495d-bf2b-66dd1927971c\") "
Sep 10 17:42:44.849748 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:44.849720    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffa5673-e74f-495d-bf2b-66dd1927971c-kube-api-access-dlft7" (OuterVolumeSpecName: "kube-api-access-dlft7") pod "fffa5673-e74f-495d-bf2b-66dd1927971c" (UID: "fffa5673-e74f-495d-bf2b-66dd1927971c"). InnerVolumeSpecName "kube-api-access-dlft7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:42:44.948798 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:44.948762    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlft7\" (UniqueName: \"kubernetes.io/projected/fffa5673-e74f-495d-bf2b-66dd1927971c-kube-api-access-dlft7\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:42:45.190615 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:45.190520    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-e15c04"]
Sep 10 17:42:45.195142 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:45.195112    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-e15c04"]
Sep 10 17:42:45.586262 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:45.586230    2552 generic.go:356] "Generic (PLEG): container finished" podID="fffa5673-e74f-495d-bf2b-66dd1927971c" containerID="00b202f5dfd7abf34fb5b0d8619c30cc556bebcc87e10252014034f87eaa2b43" exitCode=137
Sep 10 17:42:45.586679 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:45.586271    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-22c1b2"
Sep 10 17:42:45.586679 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:45.586326    2552 scope.go:117] "RemoveContainer" containerID="00b202f5dfd7abf34fb5b0d8619c30cc556bebcc87e10252014034f87eaa2b43"
Sep 10 17:42:45.586679 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:45.586575    2552 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7761-e15c04" container="test-curl-7761-e15c04"
Sep 10 17:42:45.595357 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:45.595338    2552 scope.go:117] "RemoveContainer" containerID="00b202f5dfd7abf34fb5b0d8619c30cc556bebcc87e10252014034f87eaa2b43"
Sep 10 17:42:45.595664 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:45.595646    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00b202f5dfd7abf34fb5b0d8619c30cc556bebcc87e10252014034f87eaa2b43\": container with ID starting with 00b202f5dfd7abf34fb5b0d8619c30cc556bebcc87e10252014034f87eaa2b43 not found: ID does not exist" containerID="00b202f5dfd7abf34fb5b0d8619c30cc556bebcc87e10252014034f87eaa2b43"
Sep 10 17:42:45.595716 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:45.595672    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00b202f5dfd7abf34fb5b0d8619c30cc556bebcc87e10252014034f87eaa2b43"} err="failed to get container status \"00b202f5dfd7abf34fb5b0d8619c30cc556bebcc87e10252014034f87eaa2b43\": rpc error: code = NotFound desc = could not find container \"00b202f5dfd7abf34fb5b0d8619c30cc556bebcc87e10252014034f87eaa2b43\": container with ID starting with 00b202f5dfd7abf34fb5b0d8619c30cc556bebcc87e10252014034f87eaa2b43 not found: ID does not exist"
Sep 10 17:42:46.051836 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.051749    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"]
Sep 10 17:42:46.052144 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.052129    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5cebdb0-6bc3-495a-862a-1beac529f251" containerName="test-curl-7761-e15c04"
Sep 10 17:42:46.052144 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.052144    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5cebdb0-6bc3-495a-862a-1beac529f251" containerName="test-curl-7761-e15c04"
Sep 10 17:42:46.052281 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.052214    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5cebdb0-6bc3-495a-862a-1beac529f251" containerName="test-curl-7761-e15c04"
Sep 10 17:42:46.054939 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.054918    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.057703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.057687    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1407ad96-openshift-default-dockercfg-h6qjl\""
Sep 10 17:42:46.070442 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.070416    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"]
Sep 10 17:42:46.161739 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.161663    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-credential-socket\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.161975 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.161783    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-token\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.161975 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.161816    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dndq7\" (UniqueName: \"kubernetes.io/projected/a070a9c3-26dd-4b4d-bf55-f870c92824a6-kube-api-access-dndq7\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.161975 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.161846    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-envoy\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.161975 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.161879    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istiod-ca-cert\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.162229 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.161999    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-podinfo\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.162229 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.162048    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-workload-socket\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.162229 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.162077    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-workload-certs\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.162229 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.162102    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-data\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.263586 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.263546    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-credential-socket\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.263788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.263613    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-token\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.263788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.263637    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dndq7\" (UniqueName: \"kubernetes.io/projected/a070a9c3-26dd-4b4d-bf55-f870c92824a6-kube-api-access-dndq7\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.263788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.263660    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-envoy\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.263788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.263684    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istiod-ca-cert\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.263788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.263729    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-podinfo\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.263788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.263771    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-workload-socket\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.264138 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.263802    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-workload-certs\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.264138 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.263827    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-data\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.264138 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.264039    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-credential-socket\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.264299 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.264190    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-data\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.264299 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.264222    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-workload-socket\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.264411 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.264313    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-workload-certs\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.264522 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.264467    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istiod-ca-cert\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.265936 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.265914    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-envoy\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.266272 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.266248    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-podinfo\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.273212 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.273185    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-token\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.273425 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.273402    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dndq7\" (UniqueName: \"kubernetes.io/projected/a070a9c3-26dd-4b4d-bf55-f870c92824a6-kube-api-access-dndq7\") pod \"e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") " pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.366552 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.366480    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:46.592569 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.592542    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7761-e15c04" podUID="e5cebdb0-6bc3-495a-862a-1beac529f251" containerName="test-curl-7761-e15c04" containerID="cri-o://0a44f633667c5b4416559fed03ebc8acff6426eb5bc25465fe9a922e52977741" gracePeriod=2
Sep 10 17:42:46.594651 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:46.594622    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7761-e15c04\" is forbidden: User \"system:node:ip-10-0-129-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-90.ec2.internal' and this object" podUID="e5cebdb0-6bc3-495a-862a-1beac529f251" pod="opendatahub/test-curl-7761-e15c04"
Sep 10 17:42:46.639213 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.639186    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"]
Sep 10 17:42:46.648472 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:42:46.648442    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda070a9c3_26dd_4b4d_bf55_f870c92824a6.slice/crio-82d740bf7c78a440b084c81f5a74729b353d6afb01d75040a2907c3692a63187 WatchSource:0}: Error finding container 82d740bf7c78a440b084c81f5a74729b353d6afb01d75040a2907c3692a63187: Status 404 returned error can't find the container with id 82d740bf7c78a440b084c81f5a74729b353d6afb01d75040a2907c3692a63187
Sep 10 17:42:46.650789 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.650756    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:42:46.650869 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.650821    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:42:46.650869 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.650850    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:42:46.748661 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.748628    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-0268f7"
Sep 10 17:42:46.751136 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:46.751104    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7761-e15c04\" is forbidden: User \"system:node:ip-10-0-129-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-90.ec2.internal' and this object" podUID="e5cebdb0-6bc3-495a-862a-1beac529f251" pod="opendatahub/test-curl-7761-e15c04"
Sep 10 17:42:46.808998 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.808961    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fffa5673-e74f-495d-bf2b-66dd1927971c" path="/var/lib/kubelet/pods/fffa5673-e74f-495d-bf2b-66dd1927971c/volumes"
Sep 10 17:42:46.870417 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.870376    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hltk7\" (UniqueName: \"kubernetes.io/projected/4de4d08d-bfe8-4b48-bb03-e6b90c863728-kube-api-access-hltk7\") pod \"4de4d08d-bfe8-4b48-bb03-e6b90c863728\" (UID: \"4de4d08d-bfe8-4b48-bb03-e6b90c863728\") "
Sep 10 17:42:46.873146 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.873118    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de4d08d-bfe8-4b48-bb03-e6b90c863728-kube-api-access-hltk7" (OuterVolumeSpecName: "kube-api-access-hltk7") pod "4de4d08d-bfe8-4b48-bb03-e6b90c863728" (UID: "4de4d08d-bfe8-4b48-bb03-e6b90c863728"). InnerVolumeSpecName "kube-api-access-hltk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:42:46.972368 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:46.972283    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hltk7\" (UniqueName: \"kubernetes.io/projected/4de4d08d-bfe8-4b48-bb03-e6b90c863728-kube-api-access-hltk7\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:42:47.221281 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:47.221225    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:42:47.599511 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:47.599466    2552 generic.go:356] "Generic (PLEG): container finished" podID="4de4d08d-bfe8-4b48-bb03-e6b90c863728" containerID="23d5ecd5ef9a181159dea481be6d79bab78eb57f34a4673dd08aca6eb2afb1e3" exitCode=137
Sep 10 17:42:47.599974 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:47.599550    2552 scope.go:117] "RemoveContainer" containerID="23d5ecd5ef9a181159dea481be6d79bab78eb57f34a4673dd08aca6eb2afb1e3"
Sep 10 17:42:47.599974 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:47.599554    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-0268f7"
Sep 10 17:42:47.602387 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:47.601548    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" event={"ID":"a070a9c3-26dd-4b4d-bf55-f870c92824a6","Type":"ContainerStarted","Data":"4e352173a1c93186a5853a97e2ed8106b4ccf3042708333a11e783330bf7120d"}
Sep 10 17:42:47.602387 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:47.601607    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" event={"ID":"a070a9c3-26dd-4b4d-bf55-f870c92824a6","Type":"ContainerStarted","Data":"82d740bf7c78a440b084c81f5a74729b353d6afb01d75040a2907c3692a63187"}
Sep 10 17:42:47.611337 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:47.611309    2552 scope.go:117] "RemoveContainer" containerID="23d5ecd5ef9a181159dea481be6d79bab78eb57f34a4673dd08aca6eb2afb1e3"
Sep 10 17:42:47.611702 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:47.611682    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d5ecd5ef9a181159dea481be6d79bab78eb57f34a4673dd08aca6eb2afb1e3\": container with ID starting with 23d5ecd5ef9a181159dea481be6d79bab78eb57f34a4673dd08aca6eb2afb1e3 not found: ID does not exist" containerID="23d5ecd5ef9a181159dea481be6d79bab78eb57f34a4673dd08aca6eb2afb1e3"
Sep 10 17:42:47.611754 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:47.611717    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d5ecd5ef9a181159dea481be6d79bab78eb57f34a4673dd08aca6eb2afb1e3"} err="failed to get container status \"23d5ecd5ef9a181159dea481be6d79bab78eb57f34a4673dd08aca6eb2afb1e3\": rpc error: code = NotFound desc = could not find container \"23d5ecd5ef9a181159dea481be6d79bab78eb57f34a4673dd08aca6eb2afb1e3\": container with ID starting with 23d5ecd5ef9a181159dea481be6d79bab78eb57f34a4673dd08aca6eb2afb1e3 not found: ID does not exist"
Sep 10 17:42:47.624432 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:47.624379    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" podStartSLOduration=1.624364036 podStartE2EDuration="1.624364036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:47.623251454 +0000 UTC m=+1139.381778527" watchObservedRunningTime="2026-09-10 17:42:47.624364036 +0000 UTC m=+1139.382891106"
Sep 10 17:42:48.366937 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:48.366899    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:48.371923 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:48.371900    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:48.608164 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:48.608139    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:48.609233 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:48.609215    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:42:48.781372 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:48.781342    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-e15c04"
Sep 10 17:42:48.803120 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:48.803089    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4de4d08d-bfe8-4b48-bb03-e6b90c863728" path="/var/lib/kubelet/pods/4de4d08d-bfe8-4b48-bb03-e6b90c863728/volumes"
Sep 10 17:42:48.893358 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:48.893262    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4j8l\" (UniqueName: \"kubernetes.io/projected/e5cebdb0-6bc3-495a-862a-1beac529f251-kube-api-access-g4j8l\") pod \"e5cebdb0-6bc3-495a-862a-1beac529f251\" (UID: \"e5cebdb0-6bc3-495a-862a-1beac529f251\") "
Sep 10 17:42:48.895288 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:48.895263    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cebdb0-6bc3-495a-862a-1beac529f251-kube-api-access-g4j8l" (OuterVolumeSpecName: "kube-api-access-g4j8l") pod "e5cebdb0-6bc3-495a-862a-1beac529f251" (UID: "e5cebdb0-6bc3-495a-862a-1beac529f251"). InnerVolumeSpecName "kube-api-access-g4j8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:42:48.995117 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:48.995080    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4j8l\" (UniqueName: \"kubernetes.io/projected/e5cebdb0-6bc3-495a-862a-1beac529f251-kube-api-access-g4j8l\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:42:49.614748 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:49.614713    2552 generic.go:356] "Generic (PLEG): container finished" podID="e5cebdb0-6bc3-495a-862a-1beac529f251" containerID="0a44f633667c5b4416559fed03ebc8acff6426eb5bc25465fe9a922e52977741" exitCode=137
Sep 10 17:42:49.615131 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:49.614754    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-e15c04"
Sep 10 17:42:49.615131 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:49.614816    2552 scope.go:117] "RemoveContainer" containerID="0a44f633667c5b4416559fed03ebc8acff6426eb5bc25465fe9a922e52977741"
Sep 10 17:42:49.623568 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:49.623550    2552 scope.go:117] "RemoveContainer" containerID="0a44f633667c5b4416559fed03ebc8acff6426eb5bc25465fe9a922e52977741"
Sep 10 17:42:49.623846 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:42:49.623824    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a44f633667c5b4416559fed03ebc8acff6426eb5bc25465fe9a922e52977741\": container with ID starting with 0a44f633667c5b4416559fed03ebc8acff6426eb5bc25465fe9a922e52977741 not found: ID does not exist" containerID="0a44f633667c5b4416559fed03ebc8acff6426eb5bc25465fe9a922e52977741"
Sep 10 17:42:49.623887 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:49.623854    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a44f633667c5b4416559fed03ebc8acff6426eb5bc25465fe9a922e52977741"} err="failed to get container status \"0a44f633667c5b4416559fed03ebc8acff6426eb5bc25465fe9a922e52977741\": rpc error: code = NotFound desc = could not find container \"0a44f633667c5b4416559fed03ebc8acff6426eb5bc25465fe9a922e52977741\": container with ID starting with 0a44f633667c5b4416559fed03ebc8acff6426eb5bc25465fe9a922e52977741 not found: ID does not exist"
Sep 10 17:42:50.804595 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:50.804563    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5cebdb0-6bc3-495a-862a-1beac529f251" path="/var/lib/kubelet/pods/e5cebdb0-6bc3-495a-862a-1beac529f251/volumes"
Sep 10 17:42:53.118728 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:53.118688    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:53.118728 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:53.118739    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:53.132574 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:53.132545    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:42:53.651028 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:42:53.650996    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:43:01.301510 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.301458    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"]
Sep 10 17:43:01.305354 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.305335    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.309106 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.309081    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4f14bf\"/\"openshift-service-ca.crt\""
Sep 10 17:43:01.309106 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.309106    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4f14bf\"/\"default-dockercfg-qtfxl\""
Sep 10 17:43:01.309289 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.309109    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4f14bf\"/\"auth-test-model-4f14bf-kserve-self-signed-certs\""
Sep 10 17:43:01.309289 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.309081    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4f14bf\"/\"kube-root-ca.crt\""
Sep 10 17:43:01.314333 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.314308    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"]
Sep 10 17:43:01.318018 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.317993    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-model-cache\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.318109 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.318043    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0567661a-fcc0-42d2-9dec-731b309f83bf-tls-certs\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.318109 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.318091    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-tmp-dir\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.318196 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.318121    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-home\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.318196 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.318166    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-dshm\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.318265 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.318213    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb6fv\" (UniqueName: \"kubernetes.io/projected/0567661a-fcc0-42d2-9dec-731b309f83bf-kube-api-access-hb6fv\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.419179 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.419145    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-dshm\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.419357 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.419210    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb6fv\" (UniqueName: \"kubernetes.io/projected/0567661a-fcc0-42d2-9dec-731b309f83bf-kube-api-access-hb6fv\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.419357 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.419251    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-model-cache\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.419357 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.419285    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0567661a-fcc0-42d2-9dec-731b309f83bf-tls-certs\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.419357 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.419324    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-tmp-dir\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.419357 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.419355    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-home\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.419726 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.419701    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-model-cache\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.419807 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.419748    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-tmp-dir\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.419807 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.419798    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-home\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.421512 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.421478    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-dshm\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.421845 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.421826    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0567661a-fcc0-42d2-9dec-731b309f83bf-tls-certs\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.427666 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.427644    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb6fv\" (UniqueName: \"kubernetes.io/projected/0567661a-fcc0-42d2-9dec-731b309f83bf-kube-api-access-hb6fv\") pod \"auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.615702 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.615659    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:01.915796 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:01.915770    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"]
Sep 10 17:43:01.918047 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:43:01.918017    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0567661a_fcc0_42d2_9dec_731b309f83bf.slice/crio-6ab4476a74fb34eb914639f76d101f300e2bb20ab1803d776cbb7e3351a956c6 WatchSource:0}: Error finding container 6ab4476a74fb34eb914639f76d101f300e2bb20ab1803d776cbb7e3351a956c6: Status 404 returned error can't find the container with id 6ab4476a74fb34eb914639f76d101f300e2bb20ab1803d776cbb7e3351a956c6
Sep 10 17:43:02.698223 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:02.698181    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j" event={"ID":"0567661a-fcc0-42d2-9dec-731b309f83bf","Type":"ContainerStarted","Data":"8a716624d3162eb139a7c4b2f14ec3cbf04fbda649f4ae54316880ebf27e65d9"}
Sep 10 17:43:02.698618 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:02.698239    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j" event={"ID":"0567661a-fcc0-42d2-9dec-731b309f83bf","Type":"ContainerStarted","Data":"6ab4476a74fb34eb914639f76d101f300e2bb20ab1803d776cbb7e3351a956c6"}
Sep 10 17:43:02.716213 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:02.716158    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j" podStartSLOduration=1.71614065 podStartE2EDuration="1.71614065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:43:02.714958598 +0000 UTC m=+1154.473485659" watchObservedRunningTime="2026-09-10 17:43:02.71614065 +0000 UTC m=+1154.474667720"
Sep 10 17:43:03.108747 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:03.108710    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:43:04.649685 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:04.649647    2552 patch_prober.go:28] interesting pod/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 17:43:04.650075 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:04.649716    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 17:43:06.720375 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:06.720341    2552 generic.go:356] "Generic (PLEG): container finished" podID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerID="4e352173a1c93186a5853a97e2ed8106b4ccf3042708333a11e783330bf7120d" exitCode=0
Sep 10 17:43:06.720824 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:06.720407    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" event={"ID":"a070a9c3-26dd-4b4d-bf55-f870c92824a6","Type":"ContainerDied","Data":"4e352173a1c93186a5853a97e2ed8106b4ccf3042708333a11e783330bf7120d"}
Sep 10 17:43:06.720872 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:06.720846    2552 scope.go:117] "RemoveContainer" containerID="4e352173a1c93186a5853a97e2ed8106b4ccf3042708333a11e783330bf7120d"
Sep 10 17:43:06.721946 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:06.721912    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:43:06.722050 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:06.722002    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:43:06.722112 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:06.722079    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:43:07.728086 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:07.728053    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" event={"ID":"a070a9c3-26dd-4b4d-bf55-f870c92824a6","Type":"ContainerStarted","Data":"7c0b5e07042bcf1c1034a60600db6e8d5c5cc58ce19b733ae9df8af3fae37ba0"}
Sep 10 17:43:08.367704 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:08.367661    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:43:08.367967 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:08.367879    2552 patch_prober.go:28] interesting pod/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 10 17:43:08.367967 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:08.367958    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 10 17:43:09.367753 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:09.367710    2552 patch_prober.go:28] interesting pod/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 10 17:43:09.368148 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:09.367770    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 10 17:43:10.367439 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:10.367398    2552 patch_prober.go:28] interesting pod/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 10 17:43:10.367820 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:10.367785    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 10 17:43:11.367466 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:11.367356    2552 patch_prober.go:28] interesting pod/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 10 17:43:11.367466 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:11.367432    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 10 17:43:11.616250 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:11.616215    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:11.616656 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:11.616267    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:11.628714 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:11.628638    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:11.779254 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:11.779222    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:43:12.367518 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:12.366918    2552 patch_prober.go:28] interesting pod/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 10 17:43:12.367518 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:12.366980    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 10 17:43:13.367113 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:13.367071    2552 patch_prober.go:28] interesting pod/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 10 17:43:13.368173 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:13.367723    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 10 17:43:14.367298 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:14.367256    2552 patch_prober.go:28] interesting pod/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 10 17:43:14.367922 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:14.367890    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 10 17:43:15.371271 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:15.371241    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:43:15.371662 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:15.371605    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:43:15.372360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:15.372340    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:43:16.701510 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:16.701465    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:43:33.194737 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:33.194700    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:43:48.749719 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:48.749688    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:43:48.754299 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:48.754270    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:43:48.836242 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:48.836206    2552 scope.go:117] "RemoveContainer" containerID="0bcb6d2a0d837fd63c4076151f812448c703aa21750880ac686091a118d239ab"
Sep 10 17:43:48.844948 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:48.844923    2552 scope.go:117] "RemoveContainer" containerID="f6281279c86a18dc37cbfe3f642bc53292857a5d0f6d81c25a07f0ae448baf36"
Sep 10 17:43:49.122189 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.122151    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"]
Sep 10 17:43:49.125642 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.125623    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.130411 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.130374    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1407ad96\"/\"default-dockercfg-7hfrq\""
Sep 10 17:43:49.130411 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.130394    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1407ad96\"/\"kube-root-ca.crt\""
Sep 10 17:43:49.130651 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.130432    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1407ad96\"/\"openshift-service-ca.crt\""
Sep 10 17:43:49.130651 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.130389    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1407ad96\"/\"e2e-lifecycle-model-1407ad96-kserve-self-signed-certs\""
Sep 10 17:43:49.139721 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.139696    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"]
Sep 10 17:43:49.213444 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.213408    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-dshm\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.213663 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.213453    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-tmp-dir\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.213663 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.213478    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl29s\" (UniqueName: \"kubernetes.io/projected/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-kube-api-access-pl29s\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.213663 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.213611    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-tls-certs\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.213803 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.213667    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-home\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.213847 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.213802    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-model-cache\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.315380 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.315344    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-model-cache\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.315570 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.315392    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-dshm\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.315570 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.315419    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-tmp-dir\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.315570 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.315440    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl29s\" (UniqueName: \"kubernetes.io/projected/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-kube-api-access-pl29s\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.315570 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.315470    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-tls-certs\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.315570 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.315522    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-home\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.315867 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.315836    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-model-cache\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.316001 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.315869    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-tmp-dir\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.316001 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.315924    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-home\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.317758 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.317740    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-dshm\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.318016 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.318002    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-tls-certs\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.334996 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.334972    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl29s\" (UniqueName: \"kubernetes.io/projected/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-kube-api-access-pl29s\") pod \"e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") " pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.435573 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.435461    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:49.709531 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.709478    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"]
Sep 10 17:43:49.711047 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:43:49.711017    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ed449ba_b67a_4585_8ef4_b8c5c11f4bd1.slice/crio-a9535bfc316239e0f2d916697e38434c7b87d66b957b33c8a432659816550997 WatchSource:0}: Error finding container a9535bfc316239e0f2d916697e38434c7b87d66b957b33c8a432659816550997: Status 404 returned error can't find the container with id a9535bfc316239e0f2d916697e38434c7b87d66b957b33c8a432659816550997
Sep 10 17:43:49.966811 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.966713    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz" event={"ID":"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1","Type":"ContainerStarted","Data":"0f894dfbb2a04395bb9dbe1a0360416b632655075505c2f5b9b98e363efdc8cf"}
Sep 10 17:43:49.966811 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.966764    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz" event={"ID":"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1","Type":"ContainerStarted","Data":"a9535bfc316239e0f2d916697e38434c7b87d66b957b33c8a432659816550997"}
Sep 10 17:43:49.984128 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:49.984076    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz" podStartSLOduration=0.984059869 podStartE2EDuration="984.059869ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:43:49.98327083 +0000 UTC m=+1201.741797900" watchObservedRunningTime="2026-09-10 17:43:49.984059869 +0000 UTC m=+1201.742586990"
Sep 10 17:43:50.400206 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:50.400162    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:43:59.435984 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:59.435900    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:59.436624 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:59.436005    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:43:59.457443 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:43:59.457405    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:44:00.030193 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:00.030160    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:44:07.905267 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:07.905222    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:44:14.804209 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:14.804176    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9"]
Sep 10 17:44:14.806809 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:14.806792    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9"
Sep 10 17:44:14.810135 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:14.810110    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 17:44:14.810450 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:14.810426    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:44:14.811933 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:14.811911    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9"]
Sep 10 17:44:14.870301 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:14.870260    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ca4166-ebb3-4478-b52a-d44430ec0b27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1407ad96-27rv9\" (UID: \"f6ca4166-ebb3-4478-b52a-d44430ec0b27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9"
Sep 10 17:44:14.971800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:14.971757    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ca4166-ebb3-4478-b52a-d44430ec0b27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1407ad96-27rv9\" (UID: \"f6ca4166-ebb3-4478-b52a-d44430ec0b27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9"
Sep 10 17:44:14.972402 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:14.972380    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ca4166-ebb3-4478-b52a-d44430ec0b27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1407ad96-27rv9\" (UID: \"f6ca4166-ebb3-4478-b52a-d44430ec0b27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9"
Sep 10 17:44:15.121231 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:15.121198    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9"
Sep 10 17:44:15.392592 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:15.392555    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9"]
Sep 10 17:44:15.394153 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:44:15.394119    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ca4166_ebb3_4478_b52a_d44430ec0b27.slice/crio-2cee714520314f6bae2aa15a73449418292f2d558cda62e0c00058463545b041 WatchSource:0}: Error finding container 2cee714520314f6bae2aa15a73449418292f2d558cda62e0c00058463545b041: Status 404 returned error can't find the container with id 2cee714520314f6bae2aa15a73449418292f2d558cda62e0c00058463545b041
Sep 10 17:44:16.105315 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:16.105279    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9" event={"ID":"f6ca4166-ebb3-4478-b52a-d44430ec0b27","Type":"ContainerStarted","Data":"377a6f0ab733eb22ec91f87fa686e5b57daaf716834e68f15edb57456ee5c9ea"}
Sep 10 17:44:16.105727 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:16.105345    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9" event={"ID":"f6ca4166-ebb3-4478-b52a-d44430ec0b27","Type":"ContainerStarted","Data":"2cee714520314f6bae2aa15a73449418292f2d558cda62e0c00058463545b041"}
Sep 10 17:44:16.123298 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:16.123253    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9" podStartSLOduration=2.123239621 podStartE2EDuration="2.123239621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:44:16.122936366 +0000 UTC m=+1227.881463436" watchObservedRunningTime="2026-09-10 17:44:16.123239621 +0000 UTC m=+1227.881766691"
Sep 10 17:44:17.113403 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:17.113370    2552 generic.go:356] "Generic (PLEG): container finished" podID="f6ca4166-ebb3-4478-b52a-d44430ec0b27" containerID="377a6f0ab733eb22ec91f87fa686e5b57daaf716834e68f15edb57456ee5c9ea" exitCode=0
Sep 10 17:44:17.113778 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:17.113457    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9" event={"ID":"f6ca4166-ebb3-4478-b52a-d44430ec0b27","Type":"ContainerDied","Data":"377a6f0ab733eb22ec91f87fa686e5b57daaf716834e68f15edb57456ee5c9ea"}
Sep 10 17:44:18.258720 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:18.258693    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9"
Sep 10 17:44:18.306159 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:18.306121    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ca4166-ebb3-4478-b52a-d44430ec0b27-maas-api-service-ca\") pod \"f6ca4166-ebb3-4478-b52a-d44430ec0b27\" (UID: \"f6ca4166-ebb3-4478-b52a-d44430ec0b27\") "
Sep 10 17:44:18.306472 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:18.306448    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ca4166-ebb3-4478-b52a-d44430ec0b27-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6ca4166-ebb3-4478-b52a-d44430ec0b27" (UID: "f6ca4166-ebb3-4478-b52a-d44430ec0b27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:44:18.407924 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:18.407838    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ca4166-ebb3-4478-b52a-d44430ec0b27-maas-api-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:44:19.125001 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:19.124974    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9"
Sep 10 17:44:19.125194 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:19.125013    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9" event={"ID":"f6ca4166-ebb3-4478-b52a-d44430ec0b27","Type":"ContainerDied","Data":"2cee714520314f6bae2aa15a73449418292f2d558cda62e0c00058463545b041"}
Sep 10 17:44:19.125194 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:19.125069    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cee714520314f6bae2aa15a73449418292f2d558cda62e0c00058463545b041"
Sep 10 17:44:24.997882 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:24.997843    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:44:30.673818 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:30.673783    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9"]
Sep 10 17:44:30.680618 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:30.680577    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1407ad96-27rv9"]
Sep 10 17:44:30.803168 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:30.803138    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6ca4166-ebb3-4478-b52a-d44430ec0b27" path="/var/lib/kubelet/pods/f6ca4166-ebb3-4478-b52a-d44430ec0b27/volumes"
Sep 10 17:44:38.825959 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:38.825924    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"]
Sep 10 17:44:43.407205 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:43.407112    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:44:48.872776 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:48.872737    2552 scope.go:117] "RemoveContainer" containerID="90fc4655b41ab0242751793981e63d6385307db24f22118a53f270e68a34484b"
Sep 10 17:44:48.881348 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:48.881323    2552 scope.go:117] "RemoveContainer" containerID="bb2d802b2982674d91ffd4f7d653885410a90ca41379d5d76cadca6ad32e9277"
Sep 10 17:44:48.889688 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:48.889454    2552 scope.go:117] "RemoveContainer" containerID="309384291ddac0b971c8cd353866c5e3ab0fd38eb53a6a74b7aad180af508756"
Sep 10 17:44:53.845706 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:53.845647    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz" podUID="4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1" containerName="main" containerID="cri-o://0f894dfbb2a04395bb9dbe1a0360416b632655075505c2f5b9b98e363efdc8cf" gracePeriod=45
Sep 10 17:44:54.108900 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.108837    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:44:54.177692 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.177654    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-tmp-dir\") pod \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") "
Sep 10 17:44:54.177879 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.177716    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-home\") pod \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") "
Sep 10 17:44:54.177879 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.177799    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-dshm\") pod \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") "
Sep 10 17:44:54.177879 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.177820    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-model-cache\") pod \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") "
Sep 10 17:44:54.177879 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.177867    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl29s\" (UniqueName: \"kubernetes.io/projected/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-kube-api-access-pl29s\") pod \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") "
Sep 10 17:44:54.178094 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.177894    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-tls-certs\") pod \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\" (UID: \"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1\") "
Sep 10 17:44:54.178094 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.177954    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1" (UID: "4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:44:54.178094 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.177990    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-home" (OuterVolumeSpecName: "home") pod "4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1" (UID: "4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:44:54.178252 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.178125    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1" (UID: "4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:44:54.178313 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.178297    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-model-cache\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:44:54.178359 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.178317    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-tmp-dir\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:44:54.178359 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.178330    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-home\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:44:54.180118 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.180091    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-dshm" (OuterVolumeSpecName: "dshm") pod "4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1" (UID: "4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:44:54.180376 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.180358    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1" (UID: "4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:44:54.180431 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.180372    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-kube-api-access-pl29s" (OuterVolumeSpecName: "kube-api-access-pl29s") pod "4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1" (UID: "4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1"). InnerVolumeSpecName "kube-api-access-pl29s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:44:54.279190 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.279151    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-dshm\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:44:54.279190 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.279188    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl29s\" (UniqueName: \"kubernetes.io/projected/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-kube-api-access-pl29s\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:44:54.279400 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.279202    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1-tls-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:44:54.312444 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.312415    2552 generic.go:356] "Generic (PLEG): container finished" podID="4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1" containerID="0f894dfbb2a04395bb9dbe1a0360416b632655075505c2f5b9b98e363efdc8cf" exitCode=0
Sep 10 17:44:54.312601 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.312454    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"
Sep 10 17:44:54.312601 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.312469    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz" event={"ID":"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1","Type":"ContainerDied","Data":"0f894dfbb2a04395bb9dbe1a0360416b632655075505c2f5b9b98e363efdc8cf"}
Sep 10 17:44:54.312601 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.312514    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz" event={"ID":"4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1","Type":"ContainerDied","Data":"a9535bfc316239e0f2d916697e38434c7b87d66b957b33c8a432659816550997"}
Sep 10 17:44:54.312601 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.312536    2552 scope.go:117] "RemoveContainer" containerID="0f894dfbb2a04395bb9dbe1a0360416b632655075505c2f5b9b98e363efdc8cf"
Sep 10 17:44:54.321386 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.321367    2552 scope.go:117] "RemoveContainer" containerID="0f894dfbb2a04395bb9dbe1a0360416b632655075505c2f5b9b98e363efdc8cf"
Sep 10 17:44:54.321685 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:44:54.321664    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f894dfbb2a04395bb9dbe1a0360416b632655075505c2f5b9b98e363efdc8cf\": container with ID starting with 0f894dfbb2a04395bb9dbe1a0360416b632655075505c2f5b9b98e363efdc8cf not found: ID does not exist" containerID="0f894dfbb2a04395bb9dbe1a0360416b632655075505c2f5b9b98e363efdc8cf"
Sep 10 17:44:54.321765 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.321698    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f894dfbb2a04395bb9dbe1a0360416b632655075505c2f5b9b98e363efdc8cf"} err="failed to get container status \"0f894dfbb2a04395bb9dbe1a0360416b632655075505c2f5b9b98e363efdc8cf\": rpc error: code = NotFound desc = could not find container \"0f894dfbb2a04395bb9dbe1a0360416b632655075505c2f5b9b98e363efdc8cf\": container with ID starting with 0f894dfbb2a04395bb9dbe1a0360416b632655075505c2f5b9b98e363efdc8cf not found: ID does not exist"
Sep 10 17:44:54.335139 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.335112    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"]
Sep 10 17:44:54.345239 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.345215    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-1407ad96/e2e-lifecycle-model-1407ad96-kserve-785986f559-5btpz"]
Sep 10 17:44:54.803625 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:54.803587    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1" path="/var/lib/kubelet/pods/4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1/volumes"
Sep 10 17:44:55.899463 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:44:55.899427    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:45:00.144796 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.144759    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817705-9c6s2"]
Sep 10 17:45:00.145254 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.145179    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1" containerName="main"
Sep 10 17:45:00.145254 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.145192    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1" containerName="main"
Sep 10 17:45:00.145254 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.145202    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ca4166-ebb3-4478-b52a-d44430ec0b27" containerName="revoke-keys"
Sep 10 17:45:00.145254 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.145208    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ca4166-ebb3-4478-b52a-d44430ec0b27" containerName="revoke-keys"
Sep 10 17:45:00.145391 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.145273    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6ca4166-ebb3-4478-b52a-d44430ec0b27" containerName="revoke-keys"
Sep 10 17:45:00.145391 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.145283    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ed449ba-b67a-4585-8ef4-b8c5c11f4bd1" containerName="main"
Sep 10 17:45:00.150122 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.150102    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f14bf-29817705-s6nk8"]
Sep 10 17:45:00.150224 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.150209    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-9c6s2"
Sep 10 17:45:00.153665 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.153637    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-058069-29817705-zwbjf"]
Sep 10 17:45:00.153796 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.153751    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f14bf-29817705-s6nk8"
Sep 10 17:45:00.154221 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.154199    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 17:45:00.158006 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.157985    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817705-9c6s2"]
Sep 10 17:45:00.158148 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.158128    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-058069-29817705-zwbjf"
Sep 10 17:45:00.160216 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.160185    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f14bf-29817705-s6nk8"]
Sep 10 17:45:00.162990 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.162964    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-058069-29817705-zwbjf"]
Sep 10 17:45:00.165361 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.165340    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-9c6s2"
Sep 10 17:45:00.175771 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.175753    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f14bf-29817705-s6nk8"
Sep 10 17:45:00.207803 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.207774    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-058069-29817705-zwbjf"
Sep 10 17:45:00.451889 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.451829    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817705-9c6s2"]
Sep 10 17:45:00.454320 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:45:00.454291    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d41edb2_413f_477e_bdd1_af69030b05d0.slice/crio-a9f3bf59f16479f1b3de3425306bd06af0debd356b59a3fcc317c313829fa6b9 WatchSource:0}: Error finding container a9f3bf59f16479f1b3de3425306bd06af0debd356b59a3fcc317c313829fa6b9: Status 404 returned error can't find the container with id a9f3bf59f16479f1b3de3425306bd06af0debd356b59a3fcc317c313829fa6b9
Sep 10 17:45:00.463129 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.463107    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f14bf-29817705-s6nk8"]
Sep 10 17:45:00.467193 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:45:00.467166    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79bd00d_9e4d_4e35_86dc_e5e6670e7cc1.slice/crio-48bfff8322e139b161dbdbe181aae46a44a83295248dfadc1ca304f87592bdf7 WatchSource:0}: Error finding container 48bfff8322e139b161dbdbe181aae46a44a83295248dfadc1ca304f87592bdf7: Status 404 returned error can't find the container with id 48bfff8322e139b161dbdbe181aae46a44a83295248dfadc1ca304f87592bdf7
Sep 10 17:45:00.499357 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:00.499331    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-058069-29817705-zwbjf"]
Sep 10 17:45:00.501182 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:45:00.501159    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e8e355_cb5d_4680_9c09_acd387881d02.slice/crio-657fd76a36df9ad613711f7953fbbb5d3eaf9765e63ff8d60b106ccc0d2826d4 WatchSource:0}: Error finding container 657fd76a36df9ad613711f7953fbbb5d3eaf9765e63ff8d60b106ccc0d2826d4: Status 404 returned error can't find the container with id 657fd76a36df9ad613711f7953fbbb5d3eaf9765e63ff8d60b106ccc0d2826d4
Sep 10 17:45:01.351563 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:01.351528    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-9c6s2" event={"ID":"3d41edb2-413f-477e-bdd1-af69030b05d0","Type":"ContainerStarted","Data":"b30c544f0454f380cf60456421d88da80c0fc1041cd912f855aec5dedc37f444"}
Sep 10 17:45:01.352017 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:01.351594    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-9c6s2" event={"ID":"3d41edb2-413f-477e-bdd1-af69030b05d0","Type":"ContainerStarted","Data":"a9f3bf59f16479f1b3de3425306bd06af0debd356b59a3fcc317c313829fa6b9"}
Sep 10 17:45:01.353428 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:01.353398    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f14bf-29817705-s6nk8" event={"ID":"e79bd00d-9e4d-4e35-86dc-e5e6670e7cc1","Type":"ContainerStarted","Data":"d853cf30fb9ddda19f843d56717f41bbfc46fa971118f1d858ac21f1095fd34a"}
Sep 10 17:45:01.353593 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:01.353454    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f14bf-29817705-s6nk8" event={"ID":"e79bd00d-9e4d-4e35-86dc-e5e6670e7cc1","Type":"ContainerStarted","Data":"48bfff8322e139b161dbdbe181aae46a44a83295248dfadc1ca304f87592bdf7"}
Sep 10 17:45:01.354889 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:01.354864    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-058069-29817705-zwbjf" event={"ID":"13e8e355-cb5d-4680-9c09-acd387881d02","Type":"ContainerStarted","Data":"869ebad85b248ff09d41c6af892863ce481f3b8ed76e5d02cec7a4e9fbd5f443"}
Sep 10 17:45:01.355005 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:01.354913    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-058069-29817705-zwbjf" event={"ID":"13e8e355-cb5d-4680-9c09-acd387881d02","Type":"ContainerStarted","Data":"657fd76a36df9ad613711f7953fbbb5d3eaf9765e63ff8d60b106ccc0d2826d4"}
Sep 10 17:45:01.369692 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:01.369640    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-9c6s2" podStartSLOduration=1.36962437 podStartE2EDuration="1.36962437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:45:01.367789876 +0000 UTC m=+1273.126316946" watchObservedRunningTime="2026-09-10 17:45:01.36962437 +0000 UTC m=+1273.128151442"
Sep 10 17:45:01.384193 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:01.384145    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-058069-29817705-zwbjf" podStartSLOduration=1.384129581 podStartE2EDuration="1.384129581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:45:01.382719901 +0000 UTC m=+1273.141246974" watchObservedRunningTime="2026-09-10 17:45:01.384129581 +0000 UTC m=+1273.142656651"
Sep 10 17:45:01.399123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:01.399072    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f14bf-29817705-s6nk8" podStartSLOduration=1.399056289 podStartE2EDuration="1.399056289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:45:01.397115077 +0000 UTC m=+1273.155642147" watchObservedRunningTime="2026-09-10 17:45:01.399056289 +0000 UTC m=+1273.157583359"
Sep 10 17:45:02.369155 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:02.369120    2552 generic.go:356] "Generic (PLEG): container finished" podID="13e8e355-cb5d-4680-9c09-acd387881d02" containerID="869ebad85b248ff09d41c6af892863ce481f3b8ed76e5d02cec7a4e9fbd5f443" exitCode=0
Sep 10 17:45:02.369558 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:02.369207    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-058069-29817705-zwbjf" event={"ID":"13e8e355-cb5d-4680-9c09-acd387881d02","Type":"ContainerDied","Data":"869ebad85b248ff09d41c6af892863ce481f3b8ed76e5d02cec7a4e9fbd5f443"}
Sep 10 17:45:02.370720 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:02.370698    2552 generic.go:356] "Generic (PLEG): container finished" podID="3d41edb2-413f-477e-bdd1-af69030b05d0" containerID="b30c544f0454f380cf60456421d88da80c0fc1041cd912f855aec5dedc37f444" exitCode=0
Sep 10 17:45:02.370825 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:02.370775    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-9c6s2" event={"ID":"3d41edb2-413f-477e-bdd1-af69030b05d0","Type":"ContainerDied","Data":"b30c544f0454f380cf60456421d88da80c0fc1041cd912f855aec5dedc37f444"}
Sep 10 17:45:02.372130 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:02.372109    2552 generic.go:356] "Generic (PLEG): container finished" podID="e79bd00d-9e4d-4e35-86dc-e5e6670e7cc1" containerID="d853cf30fb9ddda19f843d56717f41bbfc46fa971118f1d858ac21f1095fd34a" exitCode=0
Sep 10 17:45:02.372222 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:02.372145    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f14bf-29817705-s6nk8" event={"ID":"e79bd00d-9e4d-4e35-86dc-e5e6670e7cc1","Type":"ContainerDied","Data":"d853cf30fb9ddda19f843d56717f41bbfc46fa971118f1d858ac21f1095fd34a"}
Sep 10 17:45:03.508584 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:03.508560    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f14bf-29817705-s6nk8"
Sep 10 17:45:03.552842 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:03.552817    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-9c6s2"
Sep 10 17:45:03.580322 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:03.580294    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-058069-29817705-zwbjf"
Sep 10 17:45:04.384285 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:04.384253    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f14bf-29817705-s6nk8"
Sep 10 17:45:04.384474 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:04.384297    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f14bf-29817705-s6nk8" event={"ID":"e79bd00d-9e4d-4e35-86dc-e5e6670e7cc1","Type":"ContainerDied","Data":"48bfff8322e139b161dbdbe181aae46a44a83295248dfadc1ca304f87592bdf7"}
Sep 10 17:45:04.384474 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:04.384343    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48bfff8322e139b161dbdbe181aae46a44a83295248dfadc1ca304f87592bdf7"
Sep 10 17:45:04.385634 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:04.385615    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-058069-29817705-zwbjf"
Sep 10 17:45:04.385634 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:04.385630    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-058069-29817705-zwbjf" event={"ID":"13e8e355-cb5d-4680-9c09-acd387881d02","Type":"ContainerDied","Data":"657fd76a36df9ad613711f7953fbbb5d3eaf9765e63ff8d60b106ccc0d2826d4"}
Sep 10 17:45:04.385795 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:04.385668    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="657fd76a36df9ad613711f7953fbbb5d3eaf9765e63ff8d60b106ccc0d2826d4"
Sep 10 17:45:04.387043 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:04.387022    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-9c6s2" event={"ID":"3d41edb2-413f-477e-bdd1-af69030b05d0","Type":"ContainerDied","Data":"a9f3bf59f16479f1b3de3425306bd06af0debd356b59a3fcc317c313829fa6b9"}
Sep 10 17:45:04.387043 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:04.387034    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-9c6s2"
Sep 10 17:45:04.387179 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:04.387065    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9f3bf59f16479f1b3de3425306bd06af0debd356b59a3fcc317c313829fa6b9"
Sep 10 17:45:10.296386 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:10.296350    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:45:11.053428 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:11.053393    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"]
Sep 10 17:45:12.135341 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:12.135307    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"]
Sep 10 17:45:24.195861 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:24.195821    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:45:26.073651 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.073612    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2" podUID="bfb5e4c2-c980-49f7-bd61-23d9994f6491" containerName="main" containerID="cri-o://ebea64b73d662edf9bc7f0d35ede8d39501611b3a1dda4a56679b23f56750670" gracePeriod=45
Sep 10 17:45:26.333182 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.333153    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:45:26.411053 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.411021    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb5e4c2-c980-49f7-bd61-23d9994f6491-tls-certs\") pod \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") "
Sep 10 17:45:26.411209 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.411120    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-dshm\") pod \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") "
Sep 10 17:45:26.411209 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.411140    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-model-cache\") pod \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") "
Sep 10 17:45:26.411209 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.411182    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxpbn\" (UniqueName: \"kubernetes.io/projected/bfb5e4c2-c980-49f7-bd61-23d9994f6491-kube-api-access-mxpbn\") pod \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") "
Sep 10 17:45:26.411209 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.411202    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-home\") pod \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") "
Sep 10 17:45:26.411439 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.411225    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-tmp-dir\") pod \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\" (UID: \"bfb5e4c2-c980-49f7-bd61-23d9994f6491\") "
Sep 10 17:45:26.411527 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.411424    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-model-cache" (OuterVolumeSpecName: "model-cache") pod "bfb5e4c2-c980-49f7-bd61-23d9994f6491" (UID: "bfb5e4c2-c980-49f7-bd61-23d9994f6491"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:26.411592 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.411483    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-home" (OuterVolumeSpecName: "home") pod "bfb5e4c2-c980-49f7-bd61-23d9994f6491" (UID: "bfb5e4c2-c980-49f7-bd61-23d9994f6491"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:26.411631 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.411582    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bfb5e4c2-c980-49f7-bd61-23d9994f6491" (UID: "bfb5e4c2-c980-49f7-bd61-23d9994f6491"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:26.413456 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.413427    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb5e4c2-c980-49f7-bd61-23d9994f6491-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bfb5e4c2-c980-49f7-bd61-23d9994f6491" (UID: "bfb5e4c2-c980-49f7-bd61-23d9994f6491"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:45:26.413601 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.413505    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb5e4c2-c980-49f7-bd61-23d9994f6491-kube-api-access-mxpbn" (OuterVolumeSpecName: "kube-api-access-mxpbn") pod "bfb5e4c2-c980-49f7-bd61-23d9994f6491" (UID: "bfb5e4c2-c980-49f7-bd61-23d9994f6491"). InnerVolumeSpecName "kube-api-access-mxpbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:45:26.413601 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.413515    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-dshm" (OuterVolumeSpecName: "dshm") pod "bfb5e4c2-c980-49f7-bd61-23d9994f6491" (UID: "bfb5e4c2-c980-49f7-bd61-23d9994f6491"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:26.506563 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.506525    2552 generic.go:356] "Generic (PLEG): container finished" podID="bfb5e4c2-c980-49f7-bd61-23d9994f6491" containerID="ebea64b73d662edf9bc7f0d35ede8d39501611b3a1dda4a56679b23f56750670" exitCode=0
Sep 10 17:45:26.506709 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.506570    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"
Sep 10 17:45:26.506709 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.506611    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2" event={"ID":"bfb5e4c2-c980-49f7-bd61-23d9994f6491","Type":"ContainerDied","Data":"ebea64b73d662edf9bc7f0d35ede8d39501611b3a1dda4a56679b23f56750670"}
Sep 10 17:45:26.506709 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.506655    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2" event={"ID":"bfb5e4c2-c980-49f7-bd61-23d9994f6491","Type":"ContainerDied","Data":"409b228e4f571d357631102051b6eaf26fcb5ed9273dfd17f556ac8f8fa77ff5"}
Sep 10 17:45:26.506709 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.506676    2552 scope.go:117] "RemoveContainer" containerID="ebea64b73d662edf9bc7f0d35ede8d39501611b3a1dda4a56679b23f56750670"
Sep 10 17:45:26.512291 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.512269    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-tmp-dir\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:26.512291 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.512290    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb5e4c2-c980-49f7-bd61-23d9994f6491-tls-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:26.512438 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.512298    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-dshm\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:26.512438 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.512306    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-model-cache\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:26.512438 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.512314    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxpbn\" (UniqueName: \"kubernetes.io/projected/bfb5e4c2-c980-49f7-bd61-23d9994f6491-kube-api-access-mxpbn\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:26.512438 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.512322    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfb5e4c2-c980-49f7-bd61-23d9994f6491-home\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:26.516117 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.516097    2552 scope.go:117] "RemoveContainer" containerID="ebea64b73d662edf9bc7f0d35ede8d39501611b3a1dda4a56679b23f56750670"
Sep 10 17:45:26.516418 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:45:26.516394    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebea64b73d662edf9bc7f0d35ede8d39501611b3a1dda4a56679b23f56750670\": container with ID starting with ebea64b73d662edf9bc7f0d35ede8d39501611b3a1dda4a56679b23f56750670 not found: ID does not exist" containerID="ebea64b73d662edf9bc7f0d35ede8d39501611b3a1dda4a56679b23f56750670"
Sep 10 17:45:26.516467 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.516426    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebea64b73d662edf9bc7f0d35ede8d39501611b3a1dda4a56679b23f56750670"} err="failed to get container status \"ebea64b73d662edf9bc7f0d35ede8d39501611b3a1dda4a56679b23f56750670\": rpc error: code = NotFound desc = could not find container \"ebea64b73d662edf9bc7f0d35ede8d39501611b3a1dda4a56679b23f56750670\": container with ID starting with ebea64b73d662edf9bc7f0d35ede8d39501611b3a1dda4a56679b23f56750670 not found: ID does not exist"
Sep 10 17:45:26.528997 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.528955    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"]
Sep 10 17:45:26.532852 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.532825    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/auth-test-model-058069-kserve-8d7544c5-556z2"]
Sep 10 17:45:26.803457 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:26.803426    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfb5e4c2-c980-49f7-bd61-23d9994f6491" path="/var/lib/kubelet/pods/bfb5e4c2-c980-49f7-bd61-23d9994f6491/volumes"
Sep 10 17:45:27.155159 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.155114    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j" podUID="0567661a-fcc0-42d2-9dec-731b309f83bf" containerName="main" containerID="cri-o://8a716624d3162eb139a7c4b2f14ec3cbf04fbda649f4ae54316880ebf27e65d9" gracePeriod=45
Sep 10 17:45:27.407086 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.407017    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:45:27.514080 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.514046    2552 generic.go:356] "Generic (PLEG): container finished" podID="0567661a-fcc0-42d2-9dec-731b309f83bf" containerID="8a716624d3162eb139a7c4b2f14ec3cbf04fbda649f4ae54316880ebf27e65d9" exitCode=0
Sep 10 17:45:27.514235 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.514093    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"
Sep 10 17:45:27.514235 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.514132    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j" event={"ID":"0567661a-fcc0-42d2-9dec-731b309f83bf","Type":"ContainerDied","Data":"8a716624d3162eb139a7c4b2f14ec3cbf04fbda649f4ae54316880ebf27e65d9"}
Sep 10 17:45:27.514235 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.514175    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j" event={"ID":"0567661a-fcc0-42d2-9dec-731b309f83bf","Type":"ContainerDied","Data":"6ab4476a74fb34eb914639f76d101f300e2bb20ab1803d776cbb7e3351a956c6"}
Sep 10 17:45:27.514235 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.514193    2552 scope.go:117] "RemoveContainer" containerID="8a716624d3162eb139a7c4b2f14ec3cbf04fbda649f4ae54316880ebf27e65d9"
Sep 10 17:45:27.520801 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.520775    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-tmp-dir\") pod \"0567661a-fcc0-42d2-9dec-731b309f83bf\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") "
Sep 10 17:45:27.520933 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.520847    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0567661a-fcc0-42d2-9dec-731b309f83bf-tls-certs\") pod \"0567661a-fcc0-42d2-9dec-731b309f83bf\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") "
Sep 10 17:45:27.520998 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.520967    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb6fv\" (UniqueName: \"kubernetes.io/projected/0567661a-fcc0-42d2-9dec-731b309f83bf-kube-api-access-hb6fv\") pod \"0567661a-fcc0-42d2-9dec-731b309f83bf\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") "
Sep 10 17:45:27.521064 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.520996    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-home\") pod \"0567661a-fcc0-42d2-9dec-731b309f83bf\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") "
Sep 10 17:45:27.521064 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.521052    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-dshm\") pod \"0567661a-fcc0-42d2-9dec-731b309f83bf\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") "
Sep 10 17:45:27.521159 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.521066    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0567661a-fcc0-42d2-9dec-731b309f83bf" (UID: "0567661a-fcc0-42d2-9dec-731b309f83bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:27.521159 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.521085    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-model-cache\") pod \"0567661a-fcc0-42d2-9dec-731b309f83bf\" (UID: \"0567661a-fcc0-42d2-9dec-731b309f83bf\") "
Sep 10 17:45:27.521313 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.521288    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-home" (OuterVolumeSpecName: "home") pod "0567661a-fcc0-42d2-9dec-731b309f83bf" (UID: "0567661a-fcc0-42d2-9dec-731b309f83bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:27.521480 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.521460    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "0567661a-fcc0-42d2-9dec-731b309f83bf" (UID: "0567661a-fcc0-42d2-9dec-731b309f83bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:27.521480 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.521472    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-home\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:27.521615 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.521488    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-tmp-dir\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:27.523178 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.523140    2552 scope.go:117] "RemoveContainer" containerID="8a716624d3162eb139a7c4b2f14ec3cbf04fbda649f4ae54316880ebf27e65d9"
Sep 10 17:45:27.523262 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.523239    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0567661a-fcc0-42d2-9dec-731b309f83bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0567661a-fcc0-42d2-9dec-731b309f83bf" (UID: "0567661a-fcc0-42d2-9dec-731b309f83bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:45:27.523507 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:45:27.523470    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a716624d3162eb139a7c4b2f14ec3cbf04fbda649f4ae54316880ebf27e65d9\": container with ID starting with 8a716624d3162eb139a7c4b2f14ec3cbf04fbda649f4ae54316880ebf27e65d9 not found: ID does not exist" containerID="8a716624d3162eb139a7c4b2f14ec3cbf04fbda649f4ae54316880ebf27e65d9"
Sep 10 17:45:27.523587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.523509    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a716624d3162eb139a7c4b2f14ec3cbf04fbda649f4ae54316880ebf27e65d9"} err="failed to get container status \"8a716624d3162eb139a7c4b2f14ec3cbf04fbda649f4ae54316880ebf27e65d9\": rpc error: code = NotFound desc = could not find container \"8a716624d3162eb139a7c4b2f14ec3cbf04fbda649f4ae54316880ebf27e65d9\": container with ID starting with 8a716624d3162eb139a7c4b2f14ec3cbf04fbda649f4ae54316880ebf27e65d9 not found: ID does not exist"
Sep 10 17:45:27.523587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.523573    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-dshm" (OuterVolumeSpecName: "dshm") pod "0567661a-fcc0-42d2-9dec-731b309f83bf" (UID: "0567661a-fcc0-42d2-9dec-731b309f83bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:27.523665 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.523629    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0567661a-fcc0-42d2-9dec-731b309f83bf-kube-api-access-hb6fv" (OuterVolumeSpecName: "kube-api-access-hb6fv") pod "0567661a-fcc0-42d2-9dec-731b309f83bf" (UID: "0567661a-fcc0-42d2-9dec-731b309f83bf"). InnerVolumeSpecName "kube-api-access-hb6fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:45:27.622463 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.622424    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0567661a-fcc0-42d2-9dec-731b309f83bf-tls-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:27.622463 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.622458    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb6fv\" (UniqueName: \"kubernetes.io/projected/0567661a-fcc0-42d2-9dec-731b309f83bf-kube-api-access-hb6fv\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:27.622463 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.622468    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-dshm\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:27.622723 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.622477    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0567661a-fcc0-42d2-9dec-731b309f83bf-model-cache\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:27.837242 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.837209    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"]
Sep 10 17:45:27.841292 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:27.841266    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/auth-test-model-4f14bf-kserve-6b7db666f5-nlb4j"]
Sep 10 17:45:28.803047 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:28.803014    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0567661a-fcc0-42d2-9dec-731b309f83bf" path="/var/lib/kubelet/pods/0567661a-fcc0-42d2-9dec-731b309f83bf/volumes"
Sep 10 17:45:38.286843 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.286810    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"]
Sep 10 17:45:38.287430 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.287400    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d41edb2-413f-477e-bdd1-af69030b05d0" containerName="cleanup"
Sep 10 17:45:38.287430 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.287420    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d41edb2-413f-477e-bdd1-af69030b05d0" containerName="cleanup"
Sep 10 17:45:38.287587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.287441    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e79bd00d-9e4d-4e35-86dc-e5e6670e7cc1" containerName="cleanup"
Sep 10 17:45:38.287587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.287450    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e79bd00d-9e4d-4e35-86dc-e5e6670e7cc1" containerName="cleanup"
Sep 10 17:45:38.287587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.287476    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13e8e355-cb5d-4680-9c09-acd387881d02" containerName="cleanup"
Sep 10 17:45:38.287587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.287485    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="13e8e355-cb5d-4680-9c09-acd387881d02" containerName="cleanup"
Sep 10 17:45:38.287587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.287527    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0567661a-fcc0-42d2-9dec-731b309f83bf" containerName="main"
Sep 10 17:45:38.287587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.287537    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0567661a-fcc0-42d2-9dec-731b309f83bf" containerName="main"
Sep 10 17:45:38.287587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.287554    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb5e4c2-c980-49f7-bd61-23d9994f6491" containerName="main"
Sep 10 17:45:38.287587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.287564    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb5e4c2-c980-49f7-bd61-23d9994f6491" containerName="main"
Sep 10 17:45:38.287971 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.287665    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="13e8e355-cb5d-4680-9c09-acd387881d02" containerName="cleanup"
Sep 10 17:45:38.287971 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.287677    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="0567661a-fcc0-42d2-9dec-731b309f83bf" containerName="main"
Sep 10 17:45:38.287971 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.287688    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d41edb2-413f-477e-bdd1-af69030b05d0" containerName="cleanup"
Sep 10 17:45:38.287971 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.287702    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb5e4c2-c980-49f7-bd61-23d9994f6491" containerName="main"
Sep 10 17:45:38.287971 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.287711    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e79bd00d-9e4d-4e35-86dc-e5e6670e7cc1" containerName="cleanup"
Sep 10 17:45:38.292410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.292384    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.296328 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.296292    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-058069\"/\"kube-root-ca.crt\""
Sep 10 17:45:38.296465 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.296370    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-058069\"/\"sub-test-model-058069-kserve-self-signed-certs\""
Sep 10 17:45:38.296465 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.296398    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-058069\"/\"default-dockercfg-fz8ts\""
Sep 10 17:45:38.296465 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.296424    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-058069\"/\"openshift-service-ca.crt\""
Sep 10 17:45:38.303260 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.303237    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"]
Sep 10 17:45:38.425014 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.424977    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-home\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.425014 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.425024    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-dshm\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.425272 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.425057    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf4e747-be85-47cf-8305-de7532a62f06-tls-certs\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.425272 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.425137    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-tmp-dir\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.425272 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.425189    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5df98\" (UniqueName: \"kubernetes.io/projected/9bf4e747-be85-47cf-8305-de7532a62f06-kube-api-access-5df98\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.425397 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.425315    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-model-cache\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.526408 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.526369    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-tmp-dir\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.526408 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.526411    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5df98\" (UniqueName: \"kubernetes.io/projected/9bf4e747-be85-47cf-8305-de7532a62f06-kube-api-access-5df98\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.526675 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.526467    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-model-cache\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.526675 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.526547    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-home\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.526675 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.526567    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-dshm\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.526675 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.526581    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf4e747-be85-47cf-8305-de7532a62f06-tls-certs\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.526874 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.526857    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-tmp-dir\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.527001 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.526981    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-home\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.527070 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.527030    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-model-cache\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.528846 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.528823    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-dshm\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.529049 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.529024    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf4e747-be85-47cf-8305-de7532a62f06-tls-certs\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.535662 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.535631    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5df98\" (UniqueName: \"kubernetes.io/projected/9bf4e747-be85-47cf-8305-de7532a62f06-kube-api-access-5df98\") pod \"sub-test-model-058069-kserve-f4c897579-t5kst\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") " pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.603717 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.603677    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:38.880467 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:38.880442    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"]
Sep 10 17:45:38.882530 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:45:38.882486    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bf4e747_be85_47cf_8305_de7532a62f06.slice/crio-8a103a8975c09d0aefa739cd099440bbe636b8862870e5eb55773930b11df8cb WatchSource:0}: Error finding container 8a103a8975c09d0aefa739cd099440bbe636b8862870e5eb55773930b11df8cb: Status 404 returned error can't find the container with id 8a103a8975c09d0aefa739cd099440bbe636b8862870e5eb55773930b11df8cb
Sep 10 17:45:39.588043 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:39.587997    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst" event={"ID":"9bf4e747-be85-47cf-8305-de7532a62f06","Type":"ContainerStarted","Data":"7e683bbf167c492c98738ccee7155642b9d4271f268a0b1e8e0cfe86de19e0d5"}
Sep 10 17:45:39.588440 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:39.588053    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst" event={"ID":"9bf4e747-be85-47cf-8305-de7532a62f06","Type":"ContainerStarted","Data":"8a103a8975c09d0aefa739cd099440bbe636b8862870e5eb55773930b11df8cb"}
Sep 10 17:45:39.606983 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:39.606933    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst" podStartSLOduration=1.606917054 podStartE2EDuration="1.606917054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:45:39.605018143 +0000 UTC m=+1311.363545213" watchObservedRunningTime="2026-09-10 17:45:39.606917054 +0000 UTC m=+1311.365444124"
Sep 10 17:45:42.217270 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:42.217232    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:45:45.025983 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:45.025945    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"]
Sep 10 17:45:45.026405 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:45.026234    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy" containerID="cri-o://7c0b5e07042bcf1c1034a60600db6e8d5c5cc58ce19b733ae9df8af3fae37ba0" gracePeriod=30
Sep 10 17:45:45.374076 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:45.374035    2552 patch_prober.go:28] interesting pod/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 17:45:45.374241 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:45.374112    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 17:45:48.604538 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:48.604472    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:48.604538 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:48.604538    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:48.620379 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:48.620341    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:48.649085 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:48.649056    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:45:50.288163 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.288096    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:45:50.447114 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447078    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-credential-socket\") pod \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") "
Sep 10 17:45:50.447304 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447129    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-data\") pod \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") "
Sep 10 17:45:50.447304 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447171    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-podinfo\") pod \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") "
Sep 10 17:45:50.447304 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447207    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-workload-socket\") pod \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") "
Sep 10 17:45:50.447304 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447229    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istiod-ca-cert\") pod \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") "
Sep 10 17:45:50.447304 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447247    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-envoy\") pod \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") "
Sep 10 17:45:50.447304 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447288    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-token\") pod \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") "
Sep 10 17:45:50.447693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447322    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-workload-certs\") pod \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") "
Sep 10 17:45:50.447693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447388    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dndq7\" (UniqueName: \"kubernetes.io/projected/a070a9c3-26dd-4b4d-bf55-f870c92824a6-kube-api-access-dndq7\") pod \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\" (UID: \"a070a9c3-26dd-4b4d-bf55-f870c92824a6\") "
Sep 10 17:45:50.447693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447404    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a070a9c3-26dd-4b4d-bf55-f870c92824a6" (UID: "a070a9c3-26dd-4b4d-bf55-f870c92824a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:50.447693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447599    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-data" (OuterVolumeSpecName: "istio-data") pod "a070a9c3-26dd-4b4d-bf55-f870c92824a6" (UID: "a070a9c3-26dd-4b4d-bf55-f870c92824a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:50.447693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447616    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a070a9c3-26dd-4b4d-bf55-f870c92824a6" (UID: "a070a9c3-26dd-4b4d-bf55-f870c92824a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:50.447953 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447714    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a070a9c3-26dd-4b4d-bf55-f870c92824a6" (UID: "a070a9c3-26dd-4b4d-bf55-f870c92824a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:50.447953 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447867    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-data\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:50.447953 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447884    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-workload-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:50.447953 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447895    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-workload-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:50.447953 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447907    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-credential-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:50.447953 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.447916    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a070a9c3-26dd-4b4d-bf55-f870c92824a6" (UID: "a070a9c3-26dd-4b4d-bf55-f870c92824a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:45:50.449689 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.449660    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-token" (OuterVolumeSpecName: "istio-token") pod "a070a9c3-26dd-4b4d-bf55-f870c92824a6" (UID: "a070a9c3-26dd-4b4d-bf55-f870c92824a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:45:50.449689 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.449672    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a070a9c3-26dd-4b4d-bf55-f870c92824a6-kube-api-access-dndq7" (OuterVolumeSpecName: "kube-api-access-dndq7") pod "a070a9c3-26dd-4b4d-bf55-f870c92824a6" (UID: "a070a9c3-26dd-4b4d-bf55-f870c92824a6"). InnerVolumeSpecName "kube-api-access-dndq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:45:50.449820 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.449679    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a070a9c3-26dd-4b4d-bf55-f870c92824a6" (UID: "a070a9c3-26dd-4b4d-bf55-f870c92824a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:50.449820 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.449762    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a070a9c3-26dd-4b4d-bf55-f870c92824a6" (UID: "a070a9c3-26dd-4b4d-bf55-f870c92824a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:45:50.548861 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.548754    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dndq7\" (UniqueName: \"kubernetes.io/projected/a070a9c3-26dd-4b4d-bf55-f870c92824a6-kube-api-access-dndq7\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:50.548861 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.548795    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-podinfo\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:50.548861 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.548808    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istiod-ca-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:50.548861 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.548819    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-envoy\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:50.548861 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.548830    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a070a9c3-26dd-4b4d-bf55-f870c92824a6-istio-token\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:45:50.648955 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.648914    2552 generic.go:356] "Generic (PLEG): container finished" podID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerID="7c0b5e07042bcf1c1034a60600db6e8d5c5cc58ce19b733ae9df8af3fae37ba0" exitCode=0
Sep 10 17:45:50.649153 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.648955    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" event={"ID":"a070a9c3-26dd-4b4d-bf55-f870c92824a6","Type":"ContainerDied","Data":"7c0b5e07042bcf1c1034a60600db6e8d5c5cc58ce19b733ae9df8af3fae37ba0"}
Sep 10 17:45:50.649153 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.648970    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"
Sep 10 17:45:50.649153 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.649011    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69" event={"ID":"a070a9c3-26dd-4b4d-bf55-f870c92824a6","Type":"ContainerDied","Data":"82d740bf7c78a440b084c81f5a74729b353d6afb01d75040a2907c3692a63187"}
Sep 10 17:45:50.649153 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.649041    2552 scope.go:117] "RemoveContainer" containerID="7c0b5e07042bcf1c1034a60600db6e8d5c5cc58ce19b733ae9df8af3fae37ba0"
Sep 10 17:45:50.658724 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.658700    2552 scope.go:117] "RemoveContainer" containerID="4e352173a1c93186a5853a97e2ed8106b4ccf3042708333a11e783330bf7120d"
Sep 10 17:45:50.670856 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.670827    2552 scope.go:117] "RemoveContainer" containerID="7c0b5e07042bcf1c1034a60600db6e8d5c5cc58ce19b733ae9df8af3fae37ba0"
Sep 10 17:45:50.671223 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:45:50.671198    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c0b5e07042bcf1c1034a60600db6e8d5c5cc58ce19b733ae9df8af3fae37ba0\": container with ID starting with 7c0b5e07042bcf1c1034a60600db6e8d5c5cc58ce19b733ae9df8af3fae37ba0 not found: ID does not exist" containerID="7c0b5e07042bcf1c1034a60600db6e8d5c5cc58ce19b733ae9df8af3fae37ba0"
Sep 10 17:45:50.671409 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.671234    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0b5e07042bcf1c1034a60600db6e8d5c5cc58ce19b733ae9df8af3fae37ba0"} err="failed to get container status \"7c0b5e07042bcf1c1034a60600db6e8d5c5cc58ce19b733ae9df8af3fae37ba0\": rpc error: code = NotFound desc = could not find container \"7c0b5e07042bcf1c1034a60600db6e8d5c5cc58ce19b733ae9df8af3fae37ba0\": container with ID starting with 7c0b5e07042bcf1c1034a60600db6e8d5c5cc58ce19b733ae9df8af3fae37ba0 not found: ID does not exist"
Sep 10 17:45:50.671409 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.671253    2552 scope.go:117] "RemoveContainer" containerID="4e352173a1c93186a5853a97e2ed8106b4ccf3042708333a11e783330bf7120d"
Sep 10 17:45:50.671595 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:45:50.671569    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e352173a1c93186a5853a97e2ed8106b4ccf3042708333a11e783330bf7120d\": container with ID starting with 4e352173a1c93186a5853a97e2ed8106b4ccf3042708333a11e783330bf7120d not found: ID does not exist" containerID="4e352173a1c93186a5853a97e2ed8106b4ccf3042708333a11e783330bf7120d"
Sep 10 17:45:50.671701 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.671602    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e352173a1c93186a5853a97e2ed8106b4ccf3042708333a11e783330bf7120d"} err="failed to get container status \"4e352173a1c93186a5853a97e2ed8106b4ccf3042708333a11e783330bf7120d\": rpc error: code = NotFound desc = could not find container \"4e352173a1c93186a5853a97e2ed8106b4ccf3042708333a11e783330bf7120d\": container with ID starting with 4e352173a1c93186a5853a97e2ed8106b4ccf3042708333a11e783330bf7120d not found: ID does not exist"
Sep 10 17:45:50.673483 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.673460    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"]
Sep 10 17:45:50.677254 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.677229    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1407ad96-openshift-default-6c86bb8856-fbv69"]
Sep 10 17:45:50.803811 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:50.803732    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" path="/var/lib/kubelet/pods/a070a9c3-26dd-4b4d-bf55-f870c92824a6/volumes"
Sep 10 17:45:58.211936 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.211903    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"]
Sep 10 17:45:58.212317 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.212279    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy"
Sep 10 17:45:58.212317 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.212290    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy"
Sep 10 17:45:58.212317 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.212303    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy"
Sep 10 17:45:58.212317 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.212309    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy"
Sep 10 17:45:58.212465 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.212381    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy"
Sep 10 17:45:58.217012 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.216993    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.219598 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.219567    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-model-cache\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.219745 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.219607    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-dshm\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.219745 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.219724    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/489a6901-12f3-4f19-8b43-45524baa0fc1-tls-certs\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.219858 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.219755    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4wdp\" (UniqueName: \"kubernetes.io/projected/489a6901-12f3-4f19-8b43-45524baa0fc1-kube-api-access-h4wdp\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.219916 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.219868    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-tmp-dir\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.219974 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.219911    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-home\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.220024 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.220010    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4f14bf\"/\"openshift-service-ca.crt\""
Sep 10 17:45:58.220568 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.220549    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4f14bf\"/\"default-dockercfg-qtfxl\""
Sep 10 17:45:58.221310 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.221290    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4f14bf\"/\"kube-root-ca.crt\""
Sep 10 17:45:58.221609 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.221595    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4f14bf\"/\"sub-test-model-4f14bf-kserve-self-signed-certs\""
Sep 10 17:45:58.233115 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.233089    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"]
Sep 10 17:45:58.320465 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.320427    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-tmp-dir\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.320667 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.320475    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-home\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.320667 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.320514    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-model-cache\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.320667 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.320633    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-dshm\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.320784 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.320764    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/489a6901-12f3-4f19-8b43-45524baa0fc1-tls-certs\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.320834 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.320800    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4wdp\" (UniqueName: \"kubernetes.io/projected/489a6901-12f3-4f19-8b43-45524baa0fc1-kube-api-access-h4wdp\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.320894 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.320866    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-tmp-dir\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.320894 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.320883    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-model-cache\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.320997 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.320925    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-home\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.322897 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.322875    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-dshm\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.323186 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.323166    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/489a6901-12f3-4f19-8b43-45524baa0fc1-tls-certs\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.333900 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.333875    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4wdp\" (UniqueName: \"kubernetes.io/projected/489a6901-12f3-4f19-8b43-45524baa0fc1-kube-api-access-h4wdp\") pod \"sub-test-model-4f14bf-kserve-576b987bf-4gsb7\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.527610 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.527525    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:45:58.800774 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:45:58.800742    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod489a6901_12f3_4f19_8b43_45524baa0fc1.slice/crio-88ec380cd9bf5865d575561c2f1bc1559009b128b52cc0b70412bd826735b4bd WatchSource:0}: Error finding container 88ec380cd9bf5865d575561c2f1bc1559009b128b52cc0b70412bd826735b4bd: Status 404 returned error can't find the container with id 88ec380cd9bf5865d575561c2f1bc1559009b128b52cc0b70412bd826735b4bd
Sep 10 17:45:58.803971 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:58.803950    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"]
Sep 10 17:45:59.699298 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:59.699258    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7" event={"ID":"489a6901-12f3-4f19-8b43-45524baa0fc1","Type":"ContainerStarted","Data":"a98f717854d412d3173a6cda4712de78fae431042b7d727b25a8a074f979241a"}
Sep 10 17:45:59.699727 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:59.699308    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7" event={"ID":"489a6901-12f3-4f19-8b43-45524baa0fc1","Type":"ContainerStarted","Data":"88ec380cd9bf5865d575561c2f1bc1559009b128b52cc0b70412bd826735b4bd"}
Sep 10 17:45:59.718276 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:45:59.718216    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7" podStartSLOduration=1.718200774 podStartE2EDuration="1.718200774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:45:59.716299402 +0000 UTC m=+1331.474826473" watchObservedRunningTime="2026-09-10 17:45:59.718200774 +0000 UTC m=+1331.476727844"
Sep 10 17:46:06.407709 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:06.407668    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:46:08.528167 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:08.528126    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:46:08.528607 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:08.528232    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:46:08.540741 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:08.540710    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:46:08.757123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:08.757093    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:46:21.915386 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:21.915292    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:46:32.742457 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:32.742419    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7764-559a77"]
Sep 10 17:46:32.742905 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:32.742891    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a070a9c3-26dd-4b4d-bf55-f870c92824a6" containerName="istio-proxy"
Sep 10 17:46:32.745912 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:32.745896    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-559a77"
Sep 10 17:46:32.748631 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:32.748609    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p5m84\""
Sep 10 17:46:32.751754 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:32.751729    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7764-559a77"]
Sep 10 17:46:32.773650 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:32.773620    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnwm5\" (UniqueName: \"kubernetes.io/projected/5873ecb5-ca5d-4cd7-8e52-7f630417a4b4-kube-api-access-gnwm5\") pod \"test-curl-7764-559a77\" (UID: \"5873ecb5-ca5d-4cd7-8e52-7f630417a4b4\") " pod="opendatahub/test-curl-7764-559a77"
Sep 10 17:46:32.874731 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:32.874693    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwm5\" (UniqueName: \"kubernetes.io/projected/5873ecb5-ca5d-4cd7-8e52-7f630417a4b4-kube-api-access-gnwm5\") pod \"test-curl-7764-559a77\" (UID: \"5873ecb5-ca5d-4cd7-8e52-7f630417a4b4\") " pod="opendatahub/test-curl-7764-559a77"
Sep 10 17:46:32.883292 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:32.883257    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnwm5\" (UniqueName: \"kubernetes.io/projected/5873ecb5-ca5d-4cd7-8e52-7f630417a4b4-kube-api-access-gnwm5\") pod \"test-curl-7764-559a77\" (UID: \"5873ecb5-ca5d-4cd7-8e52-7f630417a4b4\") " pod="opendatahub/test-curl-7764-559a77"
Sep 10 17:46:33.057737 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:33.057646    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-559a77"
Sep 10 17:46:33.329480 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:33.329455    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7764-559a77"]
Sep 10 17:46:33.331516 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:46:33.331466    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5873ecb5_ca5d_4cd7_8e52_7f630417a4b4.slice/crio-3fa9069e0ada627d8c6db7eaf8a21b65882b8ca4a14496a80eb20330eb456cc6 WatchSource:0}: Error finding container 3fa9069e0ada627d8c6db7eaf8a21b65882b8ca4a14496a80eb20330eb456cc6: Status 404 returned error can't find the container with id 3fa9069e0ada627d8c6db7eaf8a21b65882b8ca4a14496a80eb20330eb456cc6
Sep 10 17:46:33.876840 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:33.876807    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7764-559a77" event={"ID":"5873ecb5-ca5d-4cd7-8e52-7f630417a4b4","Type":"ContainerStarted","Data":"96e29aaae92b9ddcf1b2ef01df38020afd8c2dda87a979392418f663e9209dda"}
Sep 10 17:46:33.876840 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:33.876846    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7764-559a77" event={"ID":"5873ecb5-ca5d-4cd7-8e52-7f630417a4b4","Type":"ContainerStarted","Data":"3fa9069e0ada627d8c6db7eaf8a21b65882b8ca4a14496a80eb20330eb456cc6"}
Sep 10 17:46:33.892759 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:33.892711    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7764-559a77" podStartSLOduration=1.892695819 podStartE2EDuration="1.892695819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:46:33.891805437 +0000 UTC m=+1365.650332508" watchObservedRunningTime="2026-09-10 17:46:33.892695819 +0000 UTC m=+1365.651222888"
Sep 10 17:46:34.489029 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:34.488981    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7764-559a77"]
Sep 10 17:46:34.491178 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:34.491147    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7764-559a77"]
Sep 10 17:46:34.882178 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:46:34.882147    2552 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7764-559a77" container="test-curl-7764-559a77"
Sep 10 17:46:35.886924 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:35.886884    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7764-559a77" podUID="5873ecb5-ca5d-4cd7-8e52-7f630417a4b4" containerName="test-curl-7764-559a77" containerID="cri-o://96e29aaae92b9ddcf1b2ef01df38020afd8c2dda87a979392418f663e9209dda" gracePeriod=2
Sep 10 17:46:38.065270 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:38.065243    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-559a77"
Sep 10 17:46:38.124905 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:38.124872    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnwm5\" (UniqueName: \"kubernetes.io/projected/5873ecb5-ca5d-4cd7-8e52-7f630417a4b4-kube-api-access-gnwm5\") pod \"5873ecb5-ca5d-4cd7-8e52-7f630417a4b4\" (UID: \"5873ecb5-ca5d-4cd7-8e52-7f630417a4b4\") "
Sep 10 17:46:38.127439 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:38.127401    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5873ecb5-ca5d-4cd7-8e52-7f630417a4b4-kube-api-access-gnwm5" (OuterVolumeSpecName: "kube-api-access-gnwm5") pod "5873ecb5-ca5d-4cd7-8e52-7f630417a4b4" (UID: "5873ecb5-ca5d-4cd7-8e52-7f630417a4b4"). InnerVolumeSpecName "kube-api-access-gnwm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:46:38.147438 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:38.147363    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"]
Sep 10 17:46:38.226327 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:38.226289    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnwm5\" (UniqueName: \"kubernetes.io/projected/5873ecb5-ca5d-4cd7-8e52-7f630417a4b4-kube-api-access-gnwm5\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:46:38.309010 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:38.308974    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:46:38.803539 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:38.803483    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5873ecb5-ca5d-4cd7-8e52-7f630417a4b4" path="/var/lib/kubelet/pods/5873ecb5-ca5d-4cd7-8e52-7f630417a4b4/volumes"
Sep 10 17:46:38.903633 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:38.903598    2552 generic.go:356] "Generic (PLEG): container finished" podID="5873ecb5-ca5d-4cd7-8e52-7f630417a4b4" containerID="96e29aaae92b9ddcf1b2ef01df38020afd8c2dda87a979392418f663e9209dda" exitCode=137
Sep 10 17:46:38.903798 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:38.903643    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-559a77"
Sep 10 17:46:38.903798 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:38.903664    2552 scope.go:117] "RemoveContainer" containerID="96e29aaae92b9ddcf1b2ef01df38020afd8c2dda87a979392418f663e9209dda"
Sep 10 17:46:38.912334 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:38.912314    2552 scope.go:117] "RemoveContainer" containerID="96e29aaae92b9ddcf1b2ef01df38020afd8c2dda87a979392418f663e9209dda"
Sep 10 17:46:38.912655 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:46:38.912636    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e29aaae92b9ddcf1b2ef01df38020afd8c2dda87a979392418f663e9209dda\": container with ID starting with 96e29aaae92b9ddcf1b2ef01df38020afd8c2dda87a979392418f663e9209dda not found: ID does not exist" containerID="96e29aaae92b9ddcf1b2ef01df38020afd8c2dda87a979392418f663e9209dda"
Sep 10 17:46:38.912710 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:38.912665    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e29aaae92b9ddcf1b2ef01df38020afd8c2dda87a979392418f663e9209dda"} err="failed to get container status \"96e29aaae92b9ddcf1b2ef01df38020afd8c2dda87a979392418f663e9209dda\": rpc error: code = NotFound desc = could not find container \"96e29aaae92b9ddcf1b2ef01df38020afd8c2dda87a979392418f663e9209dda\": container with ID starting with 96e29aaae92b9ddcf1b2ef01df38020afd8c2dda87a979392418f663e9209dda not found: ID does not exist"
Sep 10 17:46:39.164005 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:39.163969    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"]
Sep 10 17:46:40.048830 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.048795    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"]
Sep 10 17:46:40.049292 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.049271    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5873ecb5-ca5d-4cd7-8e52-7f630417a4b4" containerName="test-curl-7764-559a77"
Sep 10 17:46:40.049393 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.049293    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5873ecb5-ca5d-4cd7-8e52-7f630417a4b4" containerName="test-curl-7764-559a77"
Sep 10 17:46:40.049453 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.049428    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5873ecb5-ca5d-4cd7-8e52-7f630417a4b4" containerName="test-curl-7764-559a77"
Sep 10 17:46:40.053914 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.053892    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.056402 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.056380    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b32e7b-openshift-default-dockercfg-7lth5\""
Sep 10 17:46:40.066210 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.066184    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"]
Sep 10 17:46:40.146427 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.146390    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-token\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.146624 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.146531    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-podinfo\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.146624 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.146582    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-workload-certs\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.146624 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.146610    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-envoy\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.146778 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.146656    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-data\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.146778 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.146702    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istiod-ca-cert\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.146778 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.146722    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-credential-socket\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.146906 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.146781    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htjs4\" (UniqueName: \"kubernetes.io/projected/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-kube-api-access-htjs4\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.146906 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.146846    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-workload-socket\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.247397 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.247350    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-token\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.247889 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.247435    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-podinfo\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.247889 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.247471    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-workload-certs\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.247889 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.247530    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-envoy\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.247889 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.247547    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-data\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.247889 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.247585    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istiod-ca-cert\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.247889 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.247615    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-credential-socket\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.247889 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.247641    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htjs4\" (UniqueName: \"kubernetes.io/projected/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-kube-api-access-htjs4\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.247889 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.247671    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-workload-socket\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.248361 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.248084    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-workload-socket\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.248361 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.248150    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-workload-certs\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.248361 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.248284    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-credential-socket\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.248558 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.248452    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istiod-ca-cert\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.248622 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.248563    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-data\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.250364 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.250344    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-envoy\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.250545 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.250523    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-podinfo\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.257884 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.257860    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-token\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.257999 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.257981    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htjs4\" (UniqueName: \"kubernetes.io/projected/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-kube-api-access-htjs4\") pod \"e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") " pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.365021 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.364983    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:40.647886 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.647857    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"]
Sep 10 17:46:40.649475 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:46:40.649452    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ea8ccc_5bd6_4490_9e41_5a23e241f3e4.slice/crio-39af5c0cebc219349fabe200cf208d8718005f464403a7cdd551f46a1eb36152 WatchSource:0}: Error finding container 39af5c0cebc219349fabe200cf208d8718005f464403a7cdd551f46a1eb36152: Status 404 returned error can't find the container with id 39af5c0cebc219349fabe200cf208d8718005f464403a7cdd551f46a1eb36152
Sep 10 17:46:40.651958 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.651912    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:46:40.652063 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.652006    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:46:40.652063 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.652052    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:46:40.919274 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.919181    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm" event={"ID":"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4","Type":"ContainerStarted","Data":"35d1a487c371e2b1072523f81316e5e29b79d6b12d4ec8db28535d1af304678a"}
Sep 10 17:46:40.919274 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.919246    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm" event={"ID":"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4","Type":"ContainerStarted","Data":"39af5c0cebc219349fabe200cf208d8718005f464403a7cdd551f46a1eb36152"}
Sep 10 17:46:40.941281 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:40.941224    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm" podStartSLOduration=0.941209903 podStartE2EDuration="941.209903ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:46:40.938229832 +0000 UTC m=+1372.696756902" watchObservedRunningTime="2026-09-10 17:46:40.941209903 +0000 UTC m=+1372.699736969"
Sep 10 17:46:41.365408 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:41.365366    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:41.370426 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:41.370399    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:41.924754 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:41.924722    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:41.925914 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:41.925897    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:46:51.002874 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.002840    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"]
Sep 10 17:46:51.007756 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.007732    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.010461 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.010439    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-058069\"/\"rate-test-model-058069-kserve-self-signed-certs\""
Sep 10 17:46:51.024995 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.024965    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"]
Sep 10 17:46:51.166803 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.166763    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-dshm\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.166803 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.166803    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnlww\" (UniqueName: \"kubernetes.io/projected/f6a2f430-800c-444f-994d-d16e754856b2-kube-api-access-jnlww\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.167078 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.166832    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a2f430-800c-444f-994d-d16e754856b2-tls-certs\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.167078 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.166936    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-home\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.167078 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.166977    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-model-cache\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.167078 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.167007    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-tmp-dir\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.268168 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.268072    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-tmp-dir\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.268320 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.268209    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-dshm\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.268320 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.268229    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnlww\" (UniqueName: \"kubernetes.io/projected/f6a2f430-800c-444f-994d-d16e754856b2-kube-api-access-jnlww\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.268320 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.268253    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a2f430-800c-444f-994d-d16e754856b2-tls-certs\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.268320 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.268286    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-home\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.268320 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.268312    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-model-cache\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.268560 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.268487    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-tmp-dir\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.268682 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.268654    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-home\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.268816 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.268721    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-model-cache\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.270536 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.270489    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-dshm\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.270734 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.270717    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a2f430-800c-444f-994d-d16e754856b2-tls-certs\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.278825 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.278793    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnlww\" (UniqueName: \"kubernetes.io/projected/f6a2f430-800c-444f-994d-d16e754856b2-kube-api-access-jnlww\") pod \"rate-test-model-058069-kserve-69c74bd555-w94td\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") " pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.324688 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.324648    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:46:51.596529 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.596479    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"]
Sep 10 17:46:51.598346 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:46:51.598310    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6a2f430_800c_444f_994d_d16e754856b2.slice/crio-2d67bcc7124ce755c671c61f7440594e06285e29d0c08629ec79093a32765aca WatchSource:0}: Error finding container 2d67bcc7124ce755c671c61f7440594e06285e29d0c08629ec79093a32765aca: Status 404 returned error can't find the container with id 2d67bcc7124ce755c671c61f7440594e06285e29d0c08629ec79093a32765aca
Sep 10 17:46:51.977080 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.976977    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td" event={"ID":"f6a2f430-800c-444f-994d-d16e754856b2","Type":"ContainerStarted","Data":"c1b32a677ed5a573a93e91ae60ebf75461f6deacd1fa65c4535235f58a9958a3"}
Sep 10 17:46:51.977080 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:51.977021    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td" event={"ID":"f6a2f430-800c-444f-994d-d16e754856b2","Type":"ContainerStarted","Data":"2d67bcc7124ce755c671c61f7440594e06285e29d0c08629ec79093a32765aca"}
Sep 10 17:46:53.167435 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.167397    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst" podUID="9bf4e747-be85-47cf-8305-de7532a62f06" containerName="main" containerID="cri-o://7e683bbf167c492c98738ccee7155642b9d4271f268a0b1e8e0cfe86de19e0d5" gracePeriod=45
Sep 10 17:46:53.418639 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.418576    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:46:53.438429 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.438363    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td" podStartSLOduration=3.438347009 podStartE2EDuration="3.438347009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:46:51.995178487 +0000 UTC m=+1383.753705558" watchObservedRunningTime="2026-09-10 17:46:53.438347009 +0000 UTC m=+1385.196874079"
Sep 10 17:46:53.494170 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.494134    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf4e747-be85-47cf-8305-de7532a62f06-tls-certs\") pod \"9bf4e747-be85-47cf-8305-de7532a62f06\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") "
Sep 10 17:46:53.494370 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.494186    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-dshm\") pod \"9bf4e747-be85-47cf-8305-de7532a62f06\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") "
Sep 10 17:46:53.494370 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.494217    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5df98\" (UniqueName: \"kubernetes.io/projected/9bf4e747-be85-47cf-8305-de7532a62f06-kube-api-access-5df98\") pod \"9bf4e747-be85-47cf-8305-de7532a62f06\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") "
Sep 10 17:46:53.494370 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.494248    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-model-cache\") pod \"9bf4e747-be85-47cf-8305-de7532a62f06\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") "
Sep 10 17:46:53.494370 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.494273    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-tmp-dir\") pod \"9bf4e747-be85-47cf-8305-de7532a62f06\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") "
Sep 10 17:46:53.494370 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.494316    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-home\") pod \"9bf4e747-be85-47cf-8305-de7532a62f06\" (UID: \"9bf4e747-be85-47cf-8305-de7532a62f06\") "
Sep 10 17:46:53.494703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.494581    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9bf4e747-be85-47cf-8305-de7532a62f06" (UID: "9bf4e747-be85-47cf-8305-de7532a62f06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:46:53.494703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.494580    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-model-cache" (OuterVolumeSpecName: "model-cache") pod "9bf4e747-be85-47cf-8305-de7532a62f06" (UID: "9bf4e747-be85-47cf-8305-de7532a62f06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:46:53.494703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.494664    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-home" (OuterVolumeSpecName: "home") pod "9bf4e747-be85-47cf-8305-de7532a62f06" (UID: "9bf4e747-be85-47cf-8305-de7532a62f06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:46:53.494861 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.494737    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-model-cache\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:46:53.494861 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.494759    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-tmp-dir\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:46:53.496412 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.496385    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-dshm" (OuterVolumeSpecName: "dshm") pod "9bf4e747-be85-47cf-8305-de7532a62f06" (UID: "9bf4e747-be85-47cf-8305-de7532a62f06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:46:53.496542 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.496459    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf4e747-be85-47cf-8305-de7532a62f06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9bf4e747-be85-47cf-8305-de7532a62f06" (UID: "9bf4e747-be85-47cf-8305-de7532a62f06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:46:53.496542 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.496479    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf4e747-be85-47cf-8305-de7532a62f06-kube-api-access-5df98" (OuterVolumeSpecName: "kube-api-access-5df98") pod "9bf4e747-be85-47cf-8305-de7532a62f06" (UID: "9bf4e747-be85-47cf-8305-de7532a62f06"). InnerVolumeSpecName "kube-api-access-5df98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:46:53.595635 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.595594    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5df98\" (UniqueName: \"kubernetes.io/projected/9bf4e747-be85-47cf-8305-de7532a62f06-kube-api-access-5df98\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:46:53.595635 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.595626    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-home\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:46:53.595635 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.595642    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf4e747-be85-47cf-8305-de7532a62f06-tls-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:46:53.595864 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.595653    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf4e747-be85-47cf-8305-de7532a62f06-dshm\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:46:53.989064 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.989020    2552 generic.go:356] "Generic (PLEG): container finished" podID="9bf4e747-be85-47cf-8305-de7532a62f06" containerID="7e683bbf167c492c98738ccee7155642b9d4271f268a0b1e8e0cfe86de19e0d5" exitCode=0
Sep 10 17:46:53.989246 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.989072    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"
Sep 10 17:46:53.989246 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.989077    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst" event={"ID":"9bf4e747-be85-47cf-8305-de7532a62f06","Type":"ContainerDied","Data":"7e683bbf167c492c98738ccee7155642b9d4271f268a0b1e8e0cfe86de19e0d5"}
Sep 10 17:46:53.989246 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.989117    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst" event={"ID":"9bf4e747-be85-47cf-8305-de7532a62f06","Type":"ContainerDied","Data":"8a103a8975c09d0aefa739cd099440bbe636b8862870e5eb55773930b11df8cb"}
Sep 10 17:46:53.989246 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.989138    2552 scope.go:117] "RemoveContainer" containerID="7e683bbf167c492c98738ccee7155642b9d4271f268a0b1e8e0cfe86de19e0d5"
Sep 10 17:46:53.999159 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.999137    2552 scope.go:117] "RemoveContainer" containerID="7e683bbf167c492c98738ccee7155642b9d4271f268a0b1e8e0cfe86de19e0d5"
Sep 10 17:46:53.999451 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:46:53.999431    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e683bbf167c492c98738ccee7155642b9d4271f268a0b1e8e0cfe86de19e0d5\": container with ID starting with 7e683bbf167c492c98738ccee7155642b9d4271f268a0b1e8e0cfe86de19e0d5 not found: ID does not exist" containerID="7e683bbf167c492c98738ccee7155642b9d4271f268a0b1e8e0cfe86de19e0d5"
Sep 10 17:46:53.999542 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:53.999461    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e683bbf167c492c98738ccee7155642b9d4271f268a0b1e8e0cfe86de19e0d5"} err="failed to get container status \"7e683bbf167c492c98738ccee7155642b9d4271f268a0b1e8e0cfe86de19e0d5\": rpc error: code = NotFound desc = could not find container \"7e683bbf167c492c98738ccee7155642b9d4271f268a0b1e8e0cfe86de19e0d5\": container with ID starting with 7e683bbf167c492c98738ccee7155642b9d4271f268a0b1e8e0cfe86de19e0d5 not found: ID does not exist"
Sep 10 17:46:54.012509 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.012466    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"]
Sep 10 17:46:54.016343 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.016317    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/sub-test-model-058069-kserve-f4c897579-t5kst"]
Sep 10 17:46:54.183385 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.183345    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7" podUID="489a6901-12f3-4f19-8b43-45524baa0fc1" containerName="main" containerID="cri-o://a98f717854d412d3173a6cda4712de78fae431042b7d727b25a8a074f979241a" gracePeriod=45
Sep 10 17:46:54.447396 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.447365    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:46:54.604846 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.604818    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-model-cache\") pod \"489a6901-12f3-4f19-8b43-45524baa0fc1\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") "
Sep 10 17:46:54.605019 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.604869    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/489a6901-12f3-4f19-8b43-45524baa0fc1-tls-certs\") pod \"489a6901-12f3-4f19-8b43-45524baa0fc1\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") "
Sep 10 17:46:54.605019 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.604951    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-tmp-dir\") pod \"489a6901-12f3-4f19-8b43-45524baa0fc1\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") "
Sep 10 17:46:54.605019 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.604992    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-dshm\") pod \"489a6901-12f3-4f19-8b43-45524baa0fc1\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") "
Sep 10 17:46:54.605019 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.605017    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-home\") pod \"489a6901-12f3-4f19-8b43-45524baa0fc1\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") "
Sep 10 17:46:54.605232 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.605064    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4wdp\" (UniqueName: \"kubernetes.io/projected/489a6901-12f3-4f19-8b43-45524baa0fc1-kube-api-access-h4wdp\") pod \"489a6901-12f3-4f19-8b43-45524baa0fc1\" (UID: \"489a6901-12f3-4f19-8b43-45524baa0fc1\") "
Sep 10 17:46:54.605232 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.605175    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-model-cache" (OuterVolumeSpecName: "model-cache") pod "489a6901-12f3-4f19-8b43-45524baa0fc1" (UID: "489a6901-12f3-4f19-8b43-45524baa0fc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:46:54.605343 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.605298    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "489a6901-12f3-4f19-8b43-45524baa0fc1" (UID: "489a6901-12f3-4f19-8b43-45524baa0fc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:46:54.605343 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.605313    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-home" (OuterVolumeSpecName: "home") pod "489a6901-12f3-4f19-8b43-45524baa0fc1" (UID: "489a6901-12f3-4f19-8b43-45524baa0fc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:46:54.605518 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.605477    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-tmp-dir\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:46:54.605605 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.605521    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-home\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:46:54.605605 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.605531    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-model-cache\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:46:54.607189 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.607163    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489a6901-12f3-4f19-8b43-45524baa0fc1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "489a6901-12f3-4f19-8b43-45524baa0fc1" (UID: "489a6901-12f3-4f19-8b43-45524baa0fc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:46:54.607307 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.607167    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-dshm" (OuterVolumeSpecName: "dshm") pod "489a6901-12f3-4f19-8b43-45524baa0fc1" (UID: "489a6901-12f3-4f19-8b43-45524baa0fc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:46:54.607307 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.607220    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489a6901-12f3-4f19-8b43-45524baa0fc1-kube-api-access-h4wdp" (OuterVolumeSpecName: "kube-api-access-h4wdp") pod "489a6901-12f3-4f19-8b43-45524baa0fc1" (UID: "489a6901-12f3-4f19-8b43-45524baa0fc1"). InnerVolumeSpecName "kube-api-access-h4wdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:46:54.706529 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.706476    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4wdp\" (UniqueName: \"kubernetes.io/projected/489a6901-12f3-4f19-8b43-45524baa0fc1-kube-api-access-h4wdp\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:46:54.706529 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.706526    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/489a6901-12f3-4f19-8b43-45524baa0fc1-tls-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:46:54.706727 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.706538    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/489a6901-12f3-4f19-8b43-45524baa0fc1-dshm\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:46:54.715623 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.715596    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:46:54.812088 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.812048    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bf4e747-be85-47cf-8305-de7532a62f06" path="/var/lib/kubelet/pods/9bf4e747-be85-47cf-8305-de7532a62f06/volumes"
Sep 10 17:46:54.998161 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.998062    2552 generic.go:356] "Generic (PLEG): container finished" podID="489a6901-12f3-4f19-8b43-45524baa0fc1" containerID="a98f717854d412d3173a6cda4712de78fae431042b7d727b25a8a074f979241a" exitCode=0
Sep 10 17:46:54.998161 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.998114    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"
Sep 10 17:46:54.998161 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.998140    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7" event={"ID":"489a6901-12f3-4f19-8b43-45524baa0fc1","Type":"ContainerDied","Data":"a98f717854d412d3173a6cda4712de78fae431042b7d727b25a8a074f979241a"}
Sep 10 17:46:54.998452 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.998191    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7" event={"ID":"489a6901-12f3-4f19-8b43-45524baa0fc1","Type":"ContainerDied","Data":"88ec380cd9bf5865d575561c2f1bc1559009b128b52cc0b70412bd826735b4bd"}
Sep 10 17:46:54.998452 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:54.998209    2552 scope.go:117] "RemoveContainer" containerID="a98f717854d412d3173a6cda4712de78fae431042b7d727b25a8a074f979241a"
Sep 10 17:46:55.018973 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:55.018935    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"]
Sep 10 17:46:55.019180 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:55.019088    2552 scope.go:117] "RemoveContainer" containerID="a98f717854d412d3173a6cda4712de78fae431042b7d727b25a8a074f979241a"
Sep 10 17:46:55.020528 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:46:55.020478    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98f717854d412d3173a6cda4712de78fae431042b7d727b25a8a074f979241a\": container with ID starting with a98f717854d412d3173a6cda4712de78fae431042b7d727b25a8a074f979241a not found: ID does not exist" containerID="a98f717854d412d3173a6cda4712de78fae431042b7d727b25a8a074f979241a"
Sep 10 17:46:55.020636 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:55.020540    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98f717854d412d3173a6cda4712de78fae431042b7d727b25a8a074f979241a"} err="failed to get container status \"a98f717854d412d3173a6cda4712de78fae431042b7d727b25a8a074f979241a\": rpc error: code = NotFound desc = could not find container \"a98f717854d412d3173a6cda4712de78fae431042b7d727b25a8a074f979241a\": container with ID starting with a98f717854d412d3173a6cda4712de78fae431042b7d727b25a8a074f979241a not found: ID does not exist"
Sep 10 17:46:55.033122 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:55.033083    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/sub-test-model-4f14bf-kserve-576b987bf-4gsb7"]
Sep 10 17:46:56.804247 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:46:56.804205    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="489a6901-12f3-4f19-8b43-45524baa0fc1" path="/var/lib/kubelet/pods/489a6901-12f3-4f19-8b43-45524baa0fc1/volumes"
Sep 10 17:47:01.325435 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:01.325396    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:47:01.325904 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:01.325453    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:47:01.338604 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:01.338567    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:47:02.048618 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:02.048581    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:47:05.055970 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:05.055938    2552 generic.go:356] "Generic (PLEG): container finished" podID="f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" containerID="35d1a487c371e2b1072523f81316e5e29b79d6b12d4ec8db28535d1af304678a" exitCode=0
Sep 10 17:47:05.056392 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:05.056017    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm" event={"ID":"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4","Type":"ContainerDied","Data":"35d1a487c371e2b1072523f81316e5e29b79d6b12d4ec8db28535d1af304678a"}
Sep 10 17:47:05.056533 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:05.056514    2552 scope.go:117] "RemoveContainer" containerID="35d1a487c371e2b1072523f81316e5e29b79d6b12d4ec8db28535d1af304678a"
Sep 10 17:47:05.057640 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:05.057610    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:47:05.057701 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:05.057680    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:47:05.057737 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:05.057713    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:47:06.070683 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:06.070637    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm" event={"ID":"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4","Type":"ContainerStarted","Data":"a7890a62d9140ab77cd2f2b32e54b7eea99253ec5026b6ae2423036a7f69372f"}
Sep 10 17:47:06.365547 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:06.365487    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:47:06.365769 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:06.365564    2552 patch_prober.go:28] interesting pod/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 10 17:47:06.365769 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:06.365645    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm" podUID="f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 10 17:47:07.366139 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:07.366101    2552 patch_prober.go:28] interesting pod/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 10 17:47:07.367148 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:07.366714    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm" podUID="f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 10 17:47:08.366299 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.366248    2552 patch_prober.go:28] interesting pod/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 10 17:47:08.366831 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.366325    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm" podUID="f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 10 17:47:08.741397 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.741310    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"]
Sep 10 17:47:08.741729 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.741716    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489a6901-12f3-4f19-8b43-45524baa0fc1" containerName="main"
Sep 10 17:47:08.741788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.741730    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="489a6901-12f3-4f19-8b43-45524baa0fc1" containerName="main"
Sep 10 17:47:08.741788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.741754    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bf4e747-be85-47cf-8305-de7532a62f06" containerName="main"
Sep 10 17:47:08.741788 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.741760    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bf4e747-be85-47cf-8305-de7532a62f06" containerName="main"
Sep 10 17:47:08.741900 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.741835    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="489a6901-12f3-4f19-8b43-45524baa0fc1" containerName="main"
Sep 10 17:47:08.741900 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.741845    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bf4e747-be85-47cf-8305-de7532a62f06" containerName="main"
Sep 10 17:47:08.745822 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.745794    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.758084 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.757176    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-912593-openshift-default-dockercfg-ckxcn\""
Sep 10 17:47:08.766518 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.764064    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"]
Sep 10 17:47:08.859109 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.859030    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-workload-socket\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.859398 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.859379    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-credential-socket\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.859558 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.859543    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73873db1-1c93-47c9-9644-483b0163afb4-istio-token\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.859691 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.859675    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-istio-data\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.859815 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.859801    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73873db1-1c93-47c9-9644-483b0163afb4-istio-podinfo\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.859944 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.859932    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-workload-certs\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.860077 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.860065    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j2vt\" (UniqueName: \"kubernetes.io/projected/73873db1-1c93-47c9-9644-483b0163afb4-kube-api-access-9j2vt\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.860202 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.860189    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-istio-envoy\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.860302 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.860290    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73873db1-1c93-47c9-9644-483b0163afb4-istiod-ca-cert\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.961683 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.961640    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-istio-envoy\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.961966 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.961943    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73873db1-1c93-47c9-9644-483b0163afb4-istiod-ca-cert\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.962939 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.962917    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-workload-socket\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.963352 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.963332    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-workload-socket\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.963480 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.962803    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73873db1-1c93-47c9-9644-483b0163afb4-istiod-ca-cert\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.963716 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.963699    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-credential-socket\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.963851 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.963834    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73873db1-1c93-47c9-9644-483b0163afb4-istio-token\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.964682 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.964663    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-istio-data\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.964856 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.964837    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-istio-data\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.965302 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.965268    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73873db1-1c93-47c9-9644-483b0163afb4-istio-podinfo\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.965463 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.965448    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-workload-certs\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.965638 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.965623    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j2vt\" (UniqueName: \"kubernetes.io/projected/73873db1-1c93-47c9-9644-483b0163afb4-kube-api-access-9j2vt\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.966377 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.965965    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-workload-certs\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.966565 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.965198    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-credential-socket\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.972512 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.969001    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-istio-envoy\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.972512 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.969241    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73873db1-1c93-47c9-9644-483b0163afb4-istio-podinfo\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.976062 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.976033    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73873db1-1c93-47c9-9644-483b0163afb4-istio-token\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:08.976570 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:08.976551    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j2vt\" (UniqueName: \"kubernetes.io/projected/73873db1-1c93-47c9-9644-483b0163afb4-kube-api-access-9j2vt\") pod \"e2e-api-b-912593-openshift-default-7c76d7f464-www7c\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") " pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:09.074520 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:09.074187    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:09.365504 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:09.365446    2552 patch_prober.go:28] interesting pod/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 10 17:47:09.365803 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:09.365768    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm" podUID="f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 10 17:47:09.367884 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:09.367846    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"]
Sep 10 17:47:09.372212 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:47:09.372184    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73873db1_1c93_47c9_9644_483b0163afb4.slice/crio-82bf0dda88399cbc42da31f7ecbd121dc72c6870ac85c56daf756377262c1672 WatchSource:0}: Error finding container 82bf0dda88399cbc42da31f7ecbd121dc72c6870ac85c56daf756377262c1672: Status 404 returned error can't find the container with id 82bf0dda88399cbc42da31f7ecbd121dc72c6870ac85c56daf756377262c1672
Sep 10 17:47:09.374744 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:09.374707    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:47:09.374854 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:09.374794    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:47:09.374854 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:09.374842    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:47:10.095401 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:10.095365    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c" event={"ID":"73873db1-1c93-47c9-9644-483b0163afb4","Type":"ContainerStarted","Data":"ba2fae78344bbd75cb7ddb9b690dc67afeed36820bacc913d9be09dac820378e"}
Sep 10 17:47:10.095611 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:10.095418    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c" event={"ID":"73873db1-1c93-47c9-9644-483b0163afb4","Type":"ContainerStarted","Data":"82bf0dda88399cbc42da31f7ecbd121dc72c6870ac85c56daf756377262c1672"}
Sep 10 17:47:10.121566 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:10.121509    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c" podStartSLOduration=2.12147981 podStartE2EDuration="2.12147981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:47:10.119158351 +0000 UTC m=+1401.877685435" watchObservedRunningTime="2026-09-10 17:47:10.12147981 +0000 UTC m=+1401.880006883"
Sep 10 17:47:10.366284 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:10.365769    2552 patch_prober.go:28] interesting pod/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 10 17:47:10.366284 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:10.365840    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm" podUID="f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 10 17:47:11.075879 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:11.075829    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:11.080903 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:11.080868    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:11.100938 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:11.100899    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:11.101899 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:11.101876    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:11.426849 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:11.426820    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:47:11.427408 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:11.427387    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:47:11.428946 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:11.428924    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:47:23.606169 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:23.606132    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:47:33.223945 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:33.223909    2552 generic.go:356] "Generic (PLEG): container finished" podID="73873db1-1c93-47c9-9644-483b0163afb4" containerID="ba2fae78344bbd75cb7ddb9b690dc67afeed36820bacc913d9be09dac820378e" exitCode=0
Sep 10 17:47:33.224470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:33.223986    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c" event={"ID":"73873db1-1c93-47c9-9644-483b0163afb4","Type":"ContainerDied","Data":"ba2fae78344bbd75cb7ddb9b690dc67afeed36820bacc913d9be09dac820378e"}
Sep 10 17:47:33.224600 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:33.224582    2552 scope.go:117] "RemoveContainer" containerID="ba2fae78344bbd75cb7ddb9b690dc67afeed36820bacc913d9be09dac820378e"
Sep 10 17:47:33.225845 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:33.225814    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:47:33.225914 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:33.225884    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:47:33.225963 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:33.225918    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:47:34.233926 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:34.233884    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c" event={"ID":"73873db1-1c93-47c9-9644-483b0163afb4","Type":"ContainerStarted","Data":"d85a612a75906af6e6f0fe1d09a49d5faab7688a836144fc569318f9f7935625"}
Sep 10 17:47:34.809485 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:34.809454    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"]
Sep 10 17:47:34.811968 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:34.811949    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:34.814654 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:34.814627    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4f14bf\"/\"rate-test-model-4f14bf-kserve-self-signed-certs\""
Sep 10 17:47:34.815688 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:34.815669    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4f14bf\"/\"default-dockercfg-qtfxl\""
Sep 10 17:47:34.815810 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:34.815789    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4f14bf\"/\"kube-root-ca.crt\""
Sep 10 17:47:34.815862 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:34.815844    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4f14bf\"/\"openshift-service-ca.crt\""
Sep 10 17:47:34.825614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:34.825587    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"]
Sep 10 17:47:34.964475 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:34.964420    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-tmp-dir\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:34.964719 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:34.964504    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-dshm\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:34.964719 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:34.964566    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vql64\" (UniqueName: \"kubernetes.io/projected/aab775ea-f908-4283-b2bd-a38b7e14b116-kube-api-access-vql64\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:34.964719 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:34.964620    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab775ea-f908-4283-b2bd-a38b7e14b116-tls-certs\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:34.964719 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:34.964710    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-home\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:34.964956 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:34.964793    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-model-cache\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:35.065968 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.065851    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vql64\" (UniqueName: \"kubernetes.io/projected/aab775ea-f908-4283-b2bd-a38b7e14b116-kube-api-access-vql64\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:35.066161 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.065986    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab775ea-f908-4283-b2bd-a38b7e14b116-tls-certs\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:35.066161 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.066049    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-home\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:35.066161 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.066114    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-model-cache\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:35.066351 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.066187    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-tmp-dir\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:35.066351 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.066220    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-dshm\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:35.067065 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.067038    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-home\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:35.067882 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.067852    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-model-cache\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:35.067996 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.067905    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-tmp-dir\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:35.069149 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.069122    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-dshm\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:35.069793 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.069773    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab775ea-f908-4283-b2bd-a38b7e14b116-tls-certs\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:35.075755 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.075732    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:35.075956 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.075937    2552 patch_prober.go:28] interesting pod/e2e-api-b-912593-openshift-default-7c76d7f464-www7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 10 17:47:35.076027 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.075988    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c" podUID="73873db1-1c93-47c9-9644-483b0163afb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 10 17:47:35.079574 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.079552    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vql64\" (UniqueName: \"kubernetes.io/projected/aab775ea-f908-4283-b2bd-a38b7e14b116-kube-api-access-vql64\") pod \"rate-test-model-4f14bf-kserve-54fff4995-2w9hl\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") " pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:35.121482 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.121456    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:35.420029 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:35.420002    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"]
Sep 10 17:47:35.421844 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:47:35.421817    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaab775ea_f908_4283_b2bd_a38b7e14b116.slice/crio-efe5c70aabb55452e7d0f40168e465c4f3b1f2a8c970dbb9d5a4cb0eaa8e3486 WatchSource:0}: Error finding container efe5c70aabb55452e7d0f40168e465c4f3b1f2a8c970dbb9d5a4cb0eaa8e3486: Status 404 returned error can't find the container with id efe5c70aabb55452e7d0f40168e465c4f3b1f2a8c970dbb9d5a4cb0eaa8e3486
Sep 10 17:47:36.076202 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:36.076161    2552 patch_prober.go:28] interesting pod/e2e-api-b-912593-openshift-default-7c76d7f464-www7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 10 17:47:36.076370 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:36.076228    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c" podUID="73873db1-1c93-47c9-9644-483b0163afb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 10 17:47:36.248444 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:36.247519    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl" event={"ID":"aab775ea-f908-4283-b2bd-a38b7e14b116","Type":"ContainerStarted","Data":"70fea67fa5cf2724ed3fe91585fefb9bcba97be5ca9dde5774a129b36724f6d8"}
Sep 10 17:47:36.248444 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:36.247575    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl" event={"ID":"aab775ea-f908-4283-b2bd-a38b7e14b116","Type":"ContainerStarted","Data":"efe5c70aabb55452e7d0f40168e465c4f3b1f2a8c970dbb9d5a4cb0eaa8e3486"}
Sep 10 17:47:36.270770 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:36.270711    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl" podStartSLOduration=2.270695383 podStartE2EDuration="2.270695383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:47:36.267735751 +0000 UTC m=+1428.026262822" watchObservedRunningTime="2026-09-10 17:47:36.270695383 +0000 UTC m=+1428.029222453"
Sep 10 17:47:37.075727 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:37.075680    2552 patch_prober.go:28] interesting pod/e2e-api-b-912593-openshift-default-7c76d7f464-www7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 10 17:47:37.076125 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:37.075746    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c" podUID="73873db1-1c93-47c9-9644-483b0163afb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 10 17:47:38.076279 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:38.076235    2552 patch_prober.go:28] interesting pod/e2e-api-b-912593-openshift-default-7c76d7f464-www7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 10 17:47:38.076798 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:38.076304    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c" podUID="73873db1-1c93-47c9-9644-483b0163afb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 10 17:47:39.075410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:39.075375    2552 patch_prober.go:28] interesting pod/e2e-api-b-912593-openshift-default-7c76d7f464-www7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 10 17:47:39.075671 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:39.075531    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c" podUID="73873db1-1c93-47c9-9644-483b0163afb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 10 17:47:40.105234 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:40.105199    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:40.105854 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:40.105828    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:40.107250 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:40.107230    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:47:40.177210 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:40.177167    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl"]
Sep 10 17:47:40.179841 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:40.179824    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl"
Sep 10 17:47:40.182661 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:40.182635    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:47:40.182798 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:40.182648    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 17:47:40.188562 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:40.188386    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl"]
Sep 10 17:47:40.219263 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:40.219231    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2a2d99-265b-49bd-a37d-487459663149-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl\" (UID: \"2e2a2d99-265b-49bd-a37d-487459663149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl"
Sep 10 17:47:40.320542 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:40.320488    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2a2d99-265b-49bd-a37d-487459663149-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl\" (UID: \"2e2a2d99-265b-49bd-a37d-487459663149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl"
Sep 10 17:47:40.321209 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:40.321191    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2a2d99-265b-49bd-a37d-487459663149-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl\" (UID: \"2e2a2d99-265b-49bd-a37d-487459663149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl"
Sep 10 17:47:40.493153 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:40.493071    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl"
Sep 10 17:47:40.767957 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:40.767930    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl"]
Sep 10 17:47:40.770193 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:47:40.770169    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2a2d99_265b_49bd_a37d_487459663149.slice/crio-5f883f74eace4a2ecc94d6a47d9d2c9245e6bfb0772eb204dbec2fbbe31f2a6e WatchSource:0}: Error finding container 5f883f74eace4a2ecc94d6a47d9d2c9245e6bfb0772eb204dbec2fbbe31f2a6e: Status 404 returned error can't find the container with id 5f883f74eace4a2ecc94d6a47d9d2c9245e6bfb0772eb204dbec2fbbe31f2a6e
Sep 10 17:47:41.290939 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:41.290898    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl" event={"ID":"2e2a2d99-265b-49bd-a37d-487459663149","Type":"ContainerStarted","Data":"558b307a76feb862bedc429d38d8d8307fb743e66b79d12ec78166a7a6705cea"}
Sep 10 17:47:41.291348 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:41.290965    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl" event={"ID":"2e2a2d99-265b-49bd-a37d-487459663149","Type":"ContainerStarted","Data":"5f883f74eace4a2ecc94d6a47d9d2c9245e6bfb0772eb204dbec2fbbe31f2a6e"}
Sep 10 17:47:41.306914 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:41.306865    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl" podStartSLOduration=1.306849398 podStartE2EDuration="1.306849398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:47:41.305629984 +0000 UTC m=+1433.064157054" watchObservedRunningTime="2026-09-10 17:47:41.306849398 +0000 UTC m=+1433.065376469"
Sep 10 17:47:41.808416 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:41.808379    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:47:42.298882 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:42.298843    2552 generic.go:356] "Generic (PLEG): container finished" podID="2e2a2d99-265b-49bd-a37d-487459663149" containerID="558b307a76feb862bedc429d38d8d8307fb743e66b79d12ec78166a7a6705cea" exitCode=0
Sep 10 17:47:42.299260 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:42.298934    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl" event={"ID":"2e2a2d99-265b-49bd-a37d-487459663149","Type":"ContainerDied","Data":"558b307a76feb862bedc429d38d8d8307fb743e66b79d12ec78166a7a6705cea"}
Sep 10 17:47:43.441323 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:43.441298    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl"
Sep 10 17:47:43.553362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:43.553333    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2a2d99-265b-49bd-a37d-487459663149-maas-api-service-ca\") pod \"2e2a2d99-265b-49bd-a37d-487459663149\" (UID: \"2e2a2d99-265b-49bd-a37d-487459663149\") "
Sep 10 17:47:43.553747 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:43.553721    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2a2d99-265b-49bd-a37d-487459663149-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e2a2d99-265b-49bd-a37d-487459663149" (UID: "2e2a2d99-265b-49bd-a37d-487459663149"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:47:43.553846 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:43.553831    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2a2d99-265b-49bd-a37d-487459663149-maas-api-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:47:44.310577 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:44.310544    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl" event={"ID":"2e2a2d99-265b-49bd-a37d-487459663149","Type":"ContainerDied","Data":"5f883f74eace4a2ecc94d6a47d9d2c9245e6bfb0772eb204dbec2fbbe31f2a6e"}
Sep 10 17:47:44.310577 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:44.310577    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl"
Sep 10 17:47:44.310791 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:44.310595    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f883f74eace4a2ecc94d6a47d9d2c9245e6bfb0772eb204dbec2fbbe31f2a6e"
Sep 10 17:47:45.121862 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:45.121831    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:45.122231 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:45.121875    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:45.134315 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:45.134291    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:45.327408 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:45.327379    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:47:55.890640 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:55.890603    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl"]
Sep 10 17:47:55.896232 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:55.896199    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32e7b-pdtpl"]
Sep 10 17:47:56.804541 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:47:56.804479    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e2a2d99-265b-49bd-a37d-487459663149" path="/var/lib/kubelet/pods/2e2a2d99-265b-49bd-a37d-487459663149/volumes"
Sep 10 17:48:02.236587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:02.236550    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"]
Sep 10 17:48:02.237175 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:02.237064    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm" podUID="f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" containerName="istio-proxy" containerID="cri-o://a7890a62d9140ab77cd2f2b32e54b7eea99253ec5026b6ae2423036a7f69372f" gracePeriod=30
Sep 10 17:48:02.563959 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:02.563876    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-912593-7ndnt"]
Sep 10 17:48:02.564324 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:02.564311    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e2a2d99-265b-49bd-a37d-487459663149" containerName="revoke-keys"
Sep 10 17:48:02.564368 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:02.564326    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e2a2d99-265b-49bd-a37d-487459663149" containerName="revoke-keys"
Sep 10 17:48:02.564410 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:02.564401    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e2a2d99-265b-49bd-a37d-487459663149" containerName="revoke-keys"
Sep 10 17:48:02.567019 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:02.567001    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-912593-7ndnt"
Sep 10 17:48:02.569790 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:02.569766    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:48:02.569909 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:02.569766    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 17:48:02.575744 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:02.575717    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-912593-7ndnt"]
Sep 10 17:48:02.653773 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:02.653734    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d154fd-d1e4-4c0d-9333-04ef58051ffd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-912593-7ndnt\" (UID: \"07d154fd-d1e4-4c0d-9333-04ef58051ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-912593-7ndnt"
Sep 10 17:48:02.754992 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:02.754952    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d154fd-d1e4-4c0d-9333-04ef58051ffd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-912593-7ndnt\" (UID: \"07d154fd-d1e4-4c0d-9333-04ef58051ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-912593-7ndnt"
Sep 10 17:48:02.755729 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:02.755701    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d154fd-d1e4-4c0d-9333-04ef58051ffd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-912593-7ndnt\" (UID: \"07d154fd-d1e4-4c0d-9333-04ef58051ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-912593-7ndnt"
Sep 10 17:48:02.878793 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:02.878759    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-912593-7ndnt"
Sep 10 17:48:02.918848 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:02.918804    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:48:03.159092 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:03.159061    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-912593-7ndnt"]
Sep 10 17:48:03.161378 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:48:03.161354    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d154fd_d1e4_4c0d_9333_04ef58051ffd.slice/crio-2ce25101d65d9acd53e64577f8a697bb624752ec6f352943f34fb1572c9f5b0a WatchSource:0}: Error finding container 2ce25101d65d9acd53e64577f8a697bb624752ec6f352943f34fb1572c9f5b0a: Status 404 returned error can't find the container with id 2ce25101d65d9acd53e64577f8a697bb624752ec6f352943f34fb1572c9f5b0a
Sep 10 17:48:03.417610 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:03.417521    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-912593-7ndnt" event={"ID":"07d154fd-d1e4-4c0d-9333-04ef58051ffd","Type":"ContainerStarted","Data":"2ce25101d65d9acd53e64577f8a697bb624752ec6f352943f34fb1572c9f5b0a"}
Sep 10 17:48:04.424521 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:04.424470    2552 generic.go:356] "Generic (PLEG): container finished" podID="07d154fd-d1e4-4c0d-9333-04ef58051ffd" containerID="5c47a38b6dcd7cdfd3c65dde579abd4b231044e7ea028c691ec938ff70f02821" exitCode=0
Sep 10 17:48:04.424964 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:04.424560    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-912593-7ndnt" event={"ID":"07d154fd-d1e4-4c0d-9333-04ef58051ffd","Type":"ContainerDied","Data":"5c47a38b6dcd7cdfd3c65dde579abd4b231044e7ea028c691ec938ff70f02821"}
Sep 10 17:48:05.569428 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:05.569395    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-912593-7ndnt"
Sep 10 17:48:05.687236 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:05.687140    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d154fd-d1e4-4c0d-9333-04ef58051ffd-maas-api-service-ca\") pod \"07d154fd-d1e4-4c0d-9333-04ef58051ffd\" (UID: \"07d154fd-d1e4-4c0d-9333-04ef58051ffd\") "
Sep 10 17:48:05.687561 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:05.687536    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d154fd-d1e4-4c0d-9333-04ef58051ffd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "07d154fd-d1e4-4c0d-9333-04ef58051ffd" (UID: "07d154fd-d1e4-4c0d-9333-04ef58051ffd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:48:05.788337 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:05.788299    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d154fd-d1e4-4c0d-9333-04ef58051ffd-maas-api-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:06.438525 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:06.438478    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-912593-7ndnt"
Sep 10 17:48:06.438724 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:06.438520    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-912593-7ndnt" event={"ID":"07d154fd-d1e4-4c0d-9333-04ef58051ffd","Type":"ContainerDied","Data":"2ce25101d65d9acd53e64577f8a697bb624752ec6f352943f34fb1572c9f5b0a"}
Sep 10 17:48:06.438724 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:06.438582    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ce25101d65d9acd53e64577f8a697bb624752ec6f352943f34fb1572c9f5b0a"
Sep 10 17:48:07.450207 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.450170    2552 generic.go:356] "Generic (PLEG): container finished" podID="f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" containerID="a7890a62d9140ab77cd2f2b32e54b7eea99253ec5026b6ae2423036a7f69372f" exitCode=0
Sep 10 17:48:07.450693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.450239    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm" event={"ID":"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4","Type":"ContainerDied","Data":"a7890a62d9140ab77cd2f2b32e54b7eea99253ec5026b6ae2423036a7f69372f"}
Sep 10 17:48:07.450693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.450323    2552 scope.go:117] "RemoveContainer" containerID="35d1a487c371e2b1072523f81316e5e29b79d6b12d4ec8db28535d1af304678a"
Sep 10 17:48:07.497513 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.497475    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:48:07.609806 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.609771    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-credential-socket\") pod \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") "
Sep 10 17:48:07.610012 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.609827    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-data\") pod \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") "
Sep 10 17:48:07.610012 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.609861    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istiod-ca-cert\") pod \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") "
Sep 10 17:48:07.610012 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.609907    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-token\") pod \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") "
Sep 10 17:48:07.610012 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.609977    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-podinfo\") pod \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") "
Sep 10 17:48:07.610220 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.610024    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-workload-certs\") pod \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") "
Sep 10 17:48:07.610220 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.610073    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-envoy\") pod \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") "
Sep 10 17:48:07.610220 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.610105    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htjs4\" (UniqueName: \"kubernetes.io/projected/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-kube-api-access-htjs4\") pod \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") "
Sep 10 17:48:07.610220 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.610100    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" (UID: "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:07.610220 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.610162    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-workload-socket\") pod \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\" (UID: \"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4\") "
Sep 10 17:48:07.610715 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.610256    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" (UID: "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:07.610892 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.610842    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" (UID: "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:07.611031 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.610925    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" (UID: "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:48:07.611344 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.611290    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" (UID: "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:07.612533 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.612409    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-workload-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:07.612533 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.612442    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-workload-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:07.612533 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.612466    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-credential-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:07.614514 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.612801    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-data\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:07.614514 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.612832    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istiod-ca-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:07.614514 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.612943    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" (UID: "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:48:07.614514 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.613136    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-kube-api-access-htjs4" (OuterVolumeSpecName: "kube-api-access-htjs4") pod "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" (UID: "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4"). InnerVolumeSpecName "kube-api-access-htjs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:48:07.614514 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.613388    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" (UID: "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:48:07.614514 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.613637    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" (UID: "f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:07.714123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.714082    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-podinfo\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:07.714123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.714116    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-envoy\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:07.714123 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.714126    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htjs4\" (UniqueName: \"kubernetes.io/projected/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-kube-api-access-htjs4\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:07.714353 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:07.714134    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4-istio-token\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:08.456725 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:08.456697    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"
Sep 10 17:48:08.457137 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:08.456722    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm" event={"ID":"f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4","Type":"ContainerDied","Data":"39af5c0cebc219349fabe200cf208d8718005f464403a7cdd551f46a1eb36152"}
Sep 10 17:48:08.457137 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:08.456793    2552 scope.go:117] "RemoveContainer" containerID="a7890a62d9140ab77cd2f2b32e54b7eea99253ec5026b6ae2423036a7f69372f"
Sep 10 17:48:08.498600 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:08.498558    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"]
Sep 10 17:48:08.508129 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:08.508093    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b32e7b-openshift-default-574bf5c46-998xm"]
Sep 10 17:48:08.803877 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:08.803797    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" path="/var/lib/kubelet/pods/f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4/volumes"
Sep 10 17:48:18.508602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:18.508564    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-912593-7ndnt"]
Sep 10 17:48:18.515412 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:18.515384    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-912593-7ndnt"]
Sep 10 17:48:18.803929 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:18.803850    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07d154fd-d1e4-4c0d-9333-04ef58051ffd" path="/var/lib/kubelet/pods/07d154fd-d1e4-4c0d-9333-04ef58051ffd/volumes"
Sep 10 17:48:23.798148 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:23.798114    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:48:25.099838 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:25.099804    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"]
Sep 10 17:48:25.100305 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:25.100018    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c" podUID="73873db1-1c93-47c9-9644-483b0163afb4" containerName="istio-proxy" containerID="cri-o://d85a612a75906af6e6f0fe1d09a49d5faab7688a836144fc569318f9f7935625" gracePeriod=30
Sep 10 17:48:25.107852 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:25.107797    2552 patch_prober.go:28] interesting pod/e2e-api-b-912593-openshift-default-7c76d7f464-www7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 17:48:25.108008 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:25.107867    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c" podUID="73873db1-1c93-47c9-9644-483b0163afb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 17:48:28.189576 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:28.189540    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"]
Sep 10 17:48:29.185208 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:29.185169    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"]
Sep 10 17:48:30.360354 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.360292    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:48:30.438802 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.438764    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-istio-data\") pod \"73873db1-1c93-47c9-9644-483b0163afb4\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") "
Sep 10 17:48:30.438989 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.438857    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73873db1-1c93-47c9-9644-483b0163afb4-istio-token\") pod \"73873db1-1c93-47c9-9644-483b0163afb4\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") "
Sep 10 17:48:30.438989 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.438916    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j2vt\" (UniqueName: \"kubernetes.io/projected/73873db1-1c93-47c9-9644-483b0163afb4-kube-api-access-9j2vt\") pod \"73873db1-1c93-47c9-9644-483b0163afb4\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") "
Sep 10 17:48:30.438989 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.438984    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-credential-socket\") pod \"73873db1-1c93-47c9-9644-483b0163afb4\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") "
Sep 10 17:48:30.439164 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.439024    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-workload-socket\") pod \"73873db1-1c93-47c9-9644-483b0163afb4\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") "
Sep 10 17:48:30.439164 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.439063    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73873db1-1c93-47c9-9644-483b0163afb4-istiod-ca-cert\") pod \"73873db1-1c93-47c9-9644-483b0163afb4\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") "
Sep 10 17:48:30.439164 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.439079    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-istio-data" (OuterVolumeSpecName: "istio-data") pod "73873db1-1c93-47c9-9644-483b0163afb4" (UID: "73873db1-1c93-47c9-9644-483b0163afb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:30.439164 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.439099    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-workload-certs\") pod \"73873db1-1c93-47c9-9644-483b0163afb4\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") "
Sep 10 17:48:30.439164 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.439151    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73873db1-1c93-47c9-9644-483b0163afb4-istio-podinfo\") pod \"73873db1-1c93-47c9-9644-483b0163afb4\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") "
Sep 10 17:48:30.439430 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.439210    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-istio-envoy\") pod \"73873db1-1c93-47c9-9644-483b0163afb4\" (UID: \"73873db1-1c93-47c9-9644-483b0163afb4\") "
Sep 10 17:48:30.439430 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.439307    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "73873db1-1c93-47c9-9644-483b0163afb4" (UID: "73873db1-1c93-47c9-9644-483b0163afb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:30.439430 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.439353    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "73873db1-1c93-47c9-9644-483b0163afb4" (UID: "73873db1-1c93-47c9-9644-483b0163afb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:30.439430 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.439401    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "73873db1-1c93-47c9-9644-483b0163afb4" (UID: "73873db1-1c93-47c9-9644-483b0163afb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:30.439430 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.439423    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73873db1-1c93-47c9-9644-483b0163afb4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "73873db1-1c93-47c9-9644-483b0163afb4" (UID: "73873db1-1c93-47c9-9644-483b0163afb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:48:30.439719 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.439618    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-istio-data\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:30.439719 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.439633    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-credential-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:30.439719 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.439643    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-workload-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:30.439719 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.439651    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73873db1-1c93-47c9-9644-483b0163afb4-istiod-ca-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:30.439719 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.439659    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-workload-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:30.441361 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.441330    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73873db1-1c93-47c9-9644-483b0163afb4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "73873db1-1c93-47c9-9644-483b0163afb4" (UID: "73873db1-1c93-47c9-9644-483b0163afb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:48:30.441465 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.441379    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73873db1-1c93-47c9-9644-483b0163afb4-kube-api-access-9j2vt" (OuterVolumeSpecName: "kube-api-access-9j2vt") pod "73873db1-1c93-47c9-9644-483b0163afb4" (UID: "73873db1-1c93-47c9-9644-483b0163afb4"). InnerVolumeSpecName "kube-api-access-9j2vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:48:30.441614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.441596    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73873db1-1c93-47c9-9644-483b0163afb4-istio-token" (OuterVolumeSpecName: "istio-token") pod "73873db1-1c93-47c9-9644-483b0163afb4" (UID: "73873db1-1c93-47c9-9644-483b0163afb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:48:30.441657 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.441595    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "73873db1-1c93-47c9-9644-483b0163afb4" (UID: "73873db1-1c93-47c9-9644-483b0163afb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:30.540632 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.540592    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j2vt\" (UniqueName: \"kubernetes.io/projected/73873db1-1c93-47c9-9644-483b0163afb4-kube-api-access-9j2vt\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:30.540632 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.540624    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73873db1-1c93-47c9-9644-483b0163afb4-istio-podinfo\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:30.540632 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.540632    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73873db1-1c93-47c9-9644-483b0163afb4-istio-envoy\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:30.540632 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.540640    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73873db1-1c93-47c9-9644-483b0163afb4-istio-token\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:30.576809 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.576773    2552 generic.go:356] "Generic (PLEG): container finished" podID="73873db1-1c93-47c9-9644-483b0163afb4" containerID="d85a612a75906af6e6f0fe1d09a49d5faab7688a836144fc569318f9f7935625" exitCode=0
Sep 10 17:48:30.576987 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.576822    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"
Sep 10 17:48:30.576987 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.576851    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c" event={"ID":"73873db1-1c93-47c9-9644-483b0163afb4","Type":"ContainerDied","Data":"d85a612a75906af6e6f0fe1d09a49d5faab7688a836144fc569318f9f7935625"}
Sep 10 17:48:30.576987 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.576900    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c" event={"ID":"73873db1-1c93-47c9-9644-483b0163afb4","Type":"ContainerDied","Data":"82bf0dda88399cbc42da31f7ecbd121dc72c6870ac85c56daf756377262c1672"}
Sep 10 17:48:30.576987 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.576923    2552 scope.go:117] "RemoveContainer" containerID="d85a612a75906af6e6f0fe1d09a49d5faab7688a836144fc569318f9f7935625"
Sep 10 17:48:30.589562 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.589541    2552 scope.go:117] "RemoveContainer" containerID="ba2fae78344bbd75cb7ddb9b690dc67afeed36820bacc913d9be09dac820378e"
Sep 10 17:48:30.605643 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.605613    2552 scope.go:117] "RemoveContainer" containerID="d85a612a75906af6e6f0fe1d09a49d5faab7688a836144fc569318f9f7935625"
Sep 10 17:48:30.605728 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.605680    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"]
Sep 10 17:48:30.605964 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:48:30.605948    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d85a612a75906af6e6f0fe1d09a49d5faab7688a836144fc569318f9f7935625\": container with ID starting with d85a612a75906af6e6f0fe1d09a49d5faab7688a836144fc569318f9f7935625 not found: ID does not exist" containerID="d85a612a75906af6e6f0fe1d09a49d5faab7688a836144fc569318f9f7935625"
Sep 10 17:48:30.606011 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.605973    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85a612a75906af6e6f0fe1d09a49d5faab7688a836144fc569318f9f7935625"} err="failed to get container status \"d85a612a75906af6e6f0fe1d09a49d5faab7688a836144fc569318f9f7935625\": rpc error: code = NotFound desc = could not find container \"d85a612a75906af6e6f0fe1d09a49d5faab7688a836144fc569318f9f7935625\": container with ID starting with d85a612a75906af6e6f0fe1d09a49d5faab7688a836144fc569318f9f7935625 not found: ID does not exist"
Sep 10 17:48:30.606011 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.605991    2552 scope.go:117] "RemoveContainer" containerID="ba2fae78344bbd75cb7ddb9b690dc67afeed36820bacc913d9be09dac820378e"
Sep 10 17:48:30.606200 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:48:30.606186    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2fae78344bbd75cb7ddb9b690dc67afeed36820bacc913d9be09dac820378e\": container with ID starting with ba2fae78344bbd75cb7ddb9b690dc67afeed36820bacc913d9be09dac820378e not found: ID does not exist" containerID="ba2fae78344bbd75cb7ddb9b690dc67afeed36820bacc913d9be09dac820378e"
Sep 10 17:48:30.606240 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.606201    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2fae78344bbd75cb7ddb9b690dc67afeed36820bacc913d9be09dac820378e"} err="failed to get container status \"ba2fae78344bbd75cb7ddb9b690dc67afeed36820bacc913d9be09dac820378e\": rpc error: code = NotFound desc = could not find container \"ba2fae78344bbd75cb7ddb9b690dc67afeed36820bacc913d9be09dac820378e\": container with ID starting with ba2fae78344bbd75cb7ddb9b690dc67afeed36820bacc913d9be09dac820378e not found: ID does not exist"
Sep 10 17:48:30.609744 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.609715    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-912593-openshift-default-7c76d7f464-www7c"]
Sep 10 17:48:30.804143 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:30.804063    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73873db1-1c93-47c9-9644-483b0163afb4" path="/var/lib/kubelet/pods/73873db1-1c93-47c9-9644-483b0163afb4/volumes"
Sep 10 17:48:40.109722 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:40.109679    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:48:43.210328 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.210283    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td" podUID="f6a2f430-800c-444f-994d-d16e754856b2" containerName="main" containerID="cri-o://c1b32a677ed5a573a93e91ae60ebf75461f6deacd1fa65c4535235f58a9958a3" gracePeriod=45
Sep 10 17:48:43.330950 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.330909    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6"]
Sep 10 17:48:43.331315 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.331301    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07d154fd-d1e4-4c0d-9333-04ef58051ffd" containerName="revoke-keys"
Sep 10 17:48:43.331396 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.331318    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="07d154fd-d1e4-4c0d-9333-04ef58051ffd" containerName="revoke-keys"
Sep 10 17:48:43.331396 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.331332    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73873db1-1c93-47c9-9644-483b0163afb4" containerName="istio-proxy"
Sep 10 17:48:43.331396 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.331340    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="73873db1-1c93-47c9-9644-483b0163afb4" containerName="istio-proxy"
Sep 10 17:48:43.331396 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.331351    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" containerName="istio-proxy"
Sep 10 17:48:43.331396 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.331357    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" containerName="istio-proxy"
Sep 10 17:48:43.331396 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.331365    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" containerName="istio-proxy"
Sep 10 17:48:43.331396 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.331370    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" containerName="istio-proxy"
Sep 10 17:48:43.331396 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.331382    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73873db1-1c93-47c9-9644-483b0163afb4" containerName="istio-proxy"
Sep 10 17:48:43.331396 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.331388    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="73873db1-1c93-47c9-9644-483b0163afb4" containerName="istio-proxy"
Sep 10 17:48:43.331865 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.331461    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="07d154fd-d1e4-4c0d-9333-04ef58051ffd" containerName="revoke-keys"
Sep 10 17:48:43.331865 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.331471    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="73873db1-1c93-47c9-9644-483b0163afb4" containerName="istio-proxy"
Sep 10 17:48:43.331865 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.331480    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" containerName="istio-proxy"
Sep 10 17:48:43.331865 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.331485    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ea8ccc-5bd6-4490-9e41-5a23e241f3e4" containerName="istio-proxy"
Sep 10 17:48:43.331865 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.331507    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="73873db1-1c93-47c9-9644-483b0163afb4" containerName="istio-proxy"
Sep 10 17:48:43.334605 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.334579    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6"
Sep 10 17:48:43.337356 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.337312    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 17:48:43.337356 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.337346    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:48:43.342001 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.341975    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6"]
Sep 10 17:48:43.367379 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.367343    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04afe44a-3c57-4488-877d-760cfc4ea027-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6\" (UID: \"04afe44a-3c57-4488-877d-760cfc4ea027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6"
Sep 10 17:48:43.455835 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.455810    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:48:43.467799 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.467720    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-tmp-dir\") pod \"f6a2f430-800c-444f-994d-d16e754856b2\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") "
Sep 10 17:48:43.467799 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.467790    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnlww\" (UniqueName: \"kubernetes.io/projected/f6a2f430-800c-444f-994d-d16e754856b2-kube-api-access-jnlww\") pod \"f6a2f430-800c-444f-994d-d16e754856b2\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") "
Sep 10 17:48:43.467972 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.467826    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-model-cache\") pod \"f6a2f430-800c-444f-994d-d16e754856b2\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") "
Sep 10 17:48:43.467972 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.467870    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a2f430-800c-444f-994d-d16e754856b2-tls-certs\") pod \"f6a2f430-800c-444f-994d-d16e754856b2\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") "
Sep 10 17:48:43.467972 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.467909    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-dshm\") pod \"f6a2f430-800c-444f-994d-d16e754856b2\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") "
Sep 10 17:48:43.467972 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.467940    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-home\") pod \"f6a2f430-800c-444f-994d-d16e754856b2\" (UID: \"f6a2f430-800c-444f-994d-d16e754856b2\") "
Sep 10 17:48:43.468152 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.467985    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6a2f430-800c-444f-994d-d16e754856b2" (UID: "f6a2f430-800c-444f-994d-d16e754856b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:43.468152 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.468092    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6a2f430-800c-444f-994d-d16e754856b2" (UID: "f6a2f430-800c-444f-994d-d16e754856b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:43.468152 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.468126    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04afe44a-3c57-4488-877d-760cfc4ea027-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6\" (UID: \"04afe44a-3c57-4488-877d-760cfc4ea027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6"
Sep 10 17:48:43.468313 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.468282    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-home" (OuterVolumeSpecName: "home") pod "f6a2f430-800c-444f-994d-d16e754856b2" (UID: "f6a2f430-800c-444f-994d-d16e754856b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:43.468313 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.468292    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-model-cache\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:43.468406 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.468326    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-tmp-dir\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:43.468796 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.468778    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04afe44a-3c57-4488-877d-760cfc4ea027-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6\" (UID: \"04afe44a-3c57-4488-877d-760cfc4ea027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6"
Sep 10 17:48:43.469975 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.469952    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a2f430-800c-444f-994d-d16e754856b2-kube-api-access-jnlww" (OuterVolumeSpecName: "kube-api-access-jnlww") pod "f6a2f430-800c-444f-994d-d16e754856b2" (UID: "f6a2f430-800c-444f-994d-d16e754856b2"). InnerVolumeSpecName "kube-api-access-jnlww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:48:43.470377 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.470358    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a2f430-800c-444f-994d-d16e754856b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6a2f430-800c-444f-994d-d16e754856b2" (UID: "f6a2f430-800c-444f-994d-d16e754856b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:48:43.470447 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.470367    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-dshm" (OuterVolumeSpecName: "dshm") pod "f6a2f430-800c-444f-994d-d16e754856b2" (UID: "f6a2f430-800c-444f-994d-d16e754856b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:43.569186 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.569148    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-dshm\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:43.569186 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.569182    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a2f430-800c-444f-994d-d16e754856b2-home\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:43.569186 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.569192    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnlww\" (UniqueName: \"kubernetes.io/projected/f6a2f430-800c-444f-994d-d16e754856b2-kube-api-access-jnlww\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:43.569437 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.569202    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a2f430-800c-444f-994d-d16e754856b2-tls-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:43.647686 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.647659    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6"
Sep 10 17:48:43.650443 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.650389    2552 generic.go:356] "Generic (PLEG): container finished" podID="f6a2f430-800c-444f-994d-d16e754856b2" containerID="c1b32a677ed5a573a93e91ae60ebf75461f6deacd1fa65c4535235f58a9958a3" exitCode=0
Sep 10 17:48:43.650443 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.650424    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td" event={"ID":"f6a2f430-800c-444f-994d-d16e754856b2","Type":"ContainerDied","Data":"c1b32a677ed5a573a93e91ae60ebf75461f6deacd1fa65c4535235f58a9958a3"}
Sep 10 17:48:43.650837 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.650469    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"
Sep 10 17:48:43.650837 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.650488    2552 scope.go:117] "RemoveContainer" containerID="c1b32a677ed5a573a93e91ae60ebf75461f6deacd1fa65c4535235f58a9958a3"
Sep 10 17:48:43.650837 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.650472    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td" event={"ID":"f6a2f430-800c-444f-994d-d16e754856b2","Type":"ContainerDied","Data":"2d67bcc7124ce755c671c61f7440594e06285e29d0c08629ec79093a32765aca"}
Sep 10 17:48:43.664619 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.664595    2552 scope.go:117] "RemoveContainer" containerID="c1b32a677ed5a573a93e91ae60ebf75461f6deacd1fa65c4535235f58a9958a3"
Sep 10 17:48:43.665227 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:48:43.665185    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b32a677ed5a573a93e91ae60ebf75461f6deacd1fa65c4535235f58a9958a3\": container with ID starting with c1b32a677ed5a573a93e91ae60ebf75461f6deacd1fa65c4535235f58a9958a3 not found: ID does not exist" containerID="c1b32a677ed5a573a93e91ae60ebf75461f6deacd1fa65c4535235f58a9958a3"
Sep 10 17:48:43.665344 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.665251    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b32a677ed5a573a93e91ae60ebf75461f6deacd1fa65c4535235f58a9958a3"} err="failed to get container status \"c1b32a677ed5a573a93e91ae60ebf75461f6deacd1fa65c4535235f58a9958a3\": rpc error: code = NotFound desc = could not find container \"c1b32a677ed5a573a93e91ae60ebf75461f6deacd1fa65c4535235f58a9958a3\": container with ID starting with c1b32a677ed5a573a93e91ae60ebf75461f6deacd1fa65c4535235f58a9958a3 not found: ID does not exist"
Sep 10 17:48:43.684475 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.684446    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"]
Sep 10 17:48:43.690780 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.690743    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-058069/rate-test-model-058069-kserve-69c74bd555-w94td"]
Sep 10 17:48:43.930737 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:43.930706    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6"]
Sep 10 17:48:43.932325 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:48:43.932295    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04afe44a_3c57_4488_877d_760cfc4ea027.slice/crio-9b93b1a0ce509a09dc859a36a9b4e66802f95210ab32faa35457e311ab7189a1 WatchSource:0}: Error finding container 9b93b1a0ce509a09dc859a36a9b4e66802f95210ab32faa35457e311ab7189a1: Status 404 returned error can't find the container with id 9b93b1a0ce509a09dc859a36a9b4e66802f95210ab32faa35457e311ab7189a1
Sep 10 17:48:44.205577 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.205473    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl" podUID="aab775ea-f908-4283-b2bd-a38b7e14b116" containerName="main" containerID="cri-o://70fea67fa5cf2724ed3fe91585fefb9bcba97be5ca9dde5774a129b36724f6d8" gracePeriod=45
Sep 10 17:48:44.464668 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.464602    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:48:44.476567 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.476541    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-tmp-dir\") pod \"aab775ea-f908-4283-b2bd-a38b7e14b116\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") "
Sep 10 17:48:44.476689 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.476577    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab775ea-f908-4283-b2bd-a38b7e14b116-tls-certs\") pod \"aab775ea-f908-4283-b2bd-a38b7e14b116\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") "
Sep 10 17:48:44.476689 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.476615    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-model-cache\") pod \"aab775ea-f908-4283-b2bd-a38b7e14b116\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") "
Sep 10 17:48:44.476689 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.476642    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-dshm\") pod \"aab775ea-f908-4283-b2bd-a38b7e14b116\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") "
Sep 10 17:48:44.476689 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.476665    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vql64\" (UniqueName: \"kubernetes.io/projected/aab775ea-f908-4283-b2bd-a38b7e14b116-kube-api-access-vql64\") pod \"aab775ea-f908-4283-b2bd-a38b7e14b116\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") "
Sep 10 17:48:44.476926 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.476734    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-home\") pod \"aab775ea-f908-4283-b2bd-a38b7e14b116\" (UID: \"aab775ea-f908-4283-b2bd-a38b7e14b116\") "
Sep 10 17:48:44.476926 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.476848    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aab775ea-f908-4283-b2bd-a38b7e14b116" (UID: "aab775ea-f908-4283-b2bd-a38b7e14b116"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:44.477049 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.476937    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-model-cache" (OuterVolumeSpecName: "model-cache") pod "aab775ea-f908-4283-b2bd-a38b7e14b116" (UID: "aab775ea-f908-4283-b2bd-a38b7e14b116"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:44.477101 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.477064    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-home" (OuterVolumeSpecName: "home") pod "aab775ea-f908-4283-b2bd-a38b7e14b116" (UID: "aab775ea-f908-4283-b2bd-a38b7e14b116"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:44.477204 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.477183    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-tmp-dir\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:44.477283 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.477207    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-model-cache\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:44.477283 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.477220    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-home\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:44.478838 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.478802    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab775ea-f908-4283-b2bd-a38b7e14b116-kube-api-access-vql64" (OuterVolumeSpecName: "kube-api-access-vql64") pod "aab775ea-f908-4283-b2bd-a38b7e14b116" (UID: "aab775ea-f908-4283-b2bd-a38b7e14b116"). InnerVolumeSpecName "kube-api-access-vql64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:48:44.478838 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.478822    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-dshm" (OuterVolumeSpecName: "dshm") pod "aab775ea-f908-4283-b2bd-a38b7e14b116" (UID: "aab775ea-f908-4283-b2bd-a38b7e14b116"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:48:44.479044 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.479030    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aab775ea-f908-4283-b2bd-a38b7e14b116-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aab775ea-f908-4283-b2bd-a38b7e14b116" (UID: "aab775ea-f908-4283-b2bd-a38b7e14b116"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:48:44.578514 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.578468    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab775ea-f908-4283-b2bd-a38b7e14b116-tls-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:44.578680 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.578528    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab775ea-f908-4283-b2bd-a38b7e14b116-dshm\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:44.578680 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.578544    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vql64\" (UniqueName: \"kubernetes.io/projected/aab775ea-f908-4283-b2bd-a38b7e14b116-kube-api-access-vql64\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:44.657375 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.657344    2552 generic.go:356] "Generic (PLEG): container finished" podID="aab775ea-f908-4283-b2bd-a38b7e14b116" containerID="70fea67fa5cf2724ed3fe91585fefb9bcba97be5ca9dde5774a129b36724f6d8" exitCode=0
Sep 10 17:48:44.657546 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.657382    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"
Sep 10 17:48:44.657546 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.657424    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl" event={"ID":"aab775ea-f908-4283-b2bd-a38b7e14b116","Type":"ContainerDied","Data":"70fea67fa5cf2724ed3fe91585fefb9bcba97be5ca9dde5774a129b36724f6d8"}
Sep 10 17:48:44.657546 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.657469    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl" event={"ID":"aab775ea-f908-4283-b2bd-a38b7e14b116","Type":"ContainerDied","Data":"efe5c70aabb55452e7d0f40168e465c4f3b1f2a8c970dbb9d5a4cb0eaa8e3486"}
Sep 10 17:48:44.657546 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.657487    2552 scope.go:117] "RemoveContainer" containerID="70fea67fa5cf2724ed3fe91585fefb9bcba97be5ca9dde5774a129b36724f6d8"
Sep 10 17:48:44.660270 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.660247    2552 generic.go:356] "Generic (PLEG): container finished" podID="04afe44a-3c57-4488-877d-760cfc4ea027" containerID="7f3c43b0469a6ab6bb77368b553ca11a14a01ef5daf9d41b34d9bdbb20d8f140" exitCode=0
Sep 10 17:48:44.660406 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.660326    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6" event={"ID":"04afe44a-3c57-4488-877d-760cfc4ea027","Type":"ContainerDied","Data":"7f3c43b0469a6ab6bb77368b553ca11a14a01ef5daf9d41b34d9bdbb20d8f140"}
Sep 10 17:48:44.660406 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.660373    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6" event={"ID":"04afe44a-3c57-4488-877d-760cfc4ea027","Type":"ContainerStarted","Data":"9b93b1a0ce509a09dc859a36a9b4e66802f95210ab32faa35457e311ab7189a1"}
Sep 10 17:48:44.704941 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.704911    2552 scope.go:117] "RemoveContainer" containerID="70fea67fa5cf2724ed3fe91585fefb9bcba97be5ca9dde5774a129b36724f6d8"
Sep 10 17:48:44.705340 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:48:44.705315    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70fea67fa5cf2724ed3fe91585fefb9bcba97be5ca9dde5774a129b36724f6d8\": container with ID starting with 70fea67fa5cf2724ed3fe91585fefb9bcba97be5ca9dde5774a129b36724f6d8 not found: ID does not exist" containerID="70fea67fa5cf2724ed3fe91585fefb9bcba97be5ca9dde5774a129b36724f6d8"
Sep 10 17:48:44.705424 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.705349    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70fea67fa5cf2724ed3fe91585fefb9bcba97be5ca9dde5774a129b36724f6d8"} err="failed to get container status \"70fea67fa5cf2724ed3fe91585fefb9bcba97be5ca9dde5774a129b36724f6d8\": rpc error: code = NotFound desc = could not find container \"70fea67fa5cf2724ed3fe91585fefb9bcba97be5ca9dde5774a129b36724f6d8\": container with ID starting with 70fea67fa5cf2724ed3fe91585fefb9bcba97be5ca9dde5774a129b36724f6d8 not found: ID does not exist"
Sep 10 17:48:44.716849 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.716779    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"]
Sep 10 17:48:44.722626 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.722602    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4f14bf/rate-test-model-4f14bf-kserve-54fff4995-2w9hl"]
Sep 10 17:48:44.804023 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.803982    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aab775ea-f908-4283-b2bd-a38b7e14b116" path="/var/lib/kubelet/pods/aab775ea-f908-4283-b2bd-a38b7e14b116/volumes"
Sep 10 17:48:44.804521 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:44.804479    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6a2f430-800c-444f-994d-d16e754856b2" path="/var/lib/kubelet/pods/f6a2f430-800c-444f-994d-d16e754856b2/volumes"
Sep 10 17:48:45.800219 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:45.800192    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6"
Sep 10 17:48:45.892350 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:45.892316    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04afe44a-3c57-4488-877d-760cfc4ea027-maas-api-service-ca\") pod \"04afe44a-3c57-4488-877d-760cfc4ea027\" (UID: \"04afe44a-3c57-4488-877d-760cfc4ea027\") "
Sep 10 17:48:45.892736 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:45.892711    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04afe44a-3c57-4488-877d-760cfc4ea027-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04afe44a-3c57-4488-877d-760cfc4ea027" (UID: "04afe44a-3c57-4488-877d-760cfc4ea027"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:48:45.993267 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:45.993177    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04afe44a-3c57-4488-877d-760cfc4ea027-maas-api-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:48:46.673833 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:46.673797    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6" event={"ID":"04afe44a-3c57-4488-877d-760cfc4ea027","Type":"ContainerDied","Data":"9b93b1a0ce509a09dc859a36a9b4e66802f95210ab32faa35457e311ab7189a1"}
Sep 10 17:48:46.673833 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:46.673820    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6"
Sep 10 17:48:46.674044 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:46.673852    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b93b1a0ce509a09dc859a36a9b4e66802f95210ab32faa35457e311ab7189a1"
Sep 10 17:48:48.784479 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:48.784448    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:48:48.789263 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:48.789236    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:48:54.022563 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:54.022523    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6"]
Sep 10 17:48:54.027824 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:54.027786    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-762738-wbbb6"]
Sep 10 17:48:54.802995 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:54.802960    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04afe44a-3c57-4488-877d-760cfc4ea027" path="/var/lib/kubelet/pods/04afe44a-3c57-4488-877d-760cfc4ea027/volumes"
Sep 10 17:48:57.300966 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:57.300932    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:48:59.130467 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.130435    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"]
Sep 10 17:48:59.131018 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.131003    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04afe44a-3c57-4488-877d-760cfc4ea027" containerName="revoke-keys"
Sep 10 17:48:59.131080 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.131022    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="04afe44a-3c57-4488-877d-760cfc4ea027" containerName="revoke-keys"
Sep 10 17:48:59.131080 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.131046    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a2f430-800c-444f-994d-d16e754856b2" containerName="main"
Sep 10 17:48:59.131080 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.131055    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a2f430-800c-444f-994d-d16e754856b2" containerName="main"
Sep 10 17:48:59.131080 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.131064    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aab775ea-f908-4283-b2bd-a38b7e14b116" containerName="main"
Sep 10 17:48:59.131080 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.131072    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="aab775ea-f908-4283-b2bd-a38b7e14b116" containerName="main"
Sep 10 17:48:59.131244 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.131164    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="04afe44a-3c57-4488-877d-760cfc4ea027" containerName="revoke-keys"
Sep 10 17:48:59.131244 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.131178    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="aab775ea-f908-4283-b2bd-a38b7e14b116" containerName="main"
Sep 10 17:48:59.131244 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.131192    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a2f430-800c-444f-994d-d16e754856b2" containerName="main"
Sep 10 17:48:59.134739 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.134715    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.138233 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.138210    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7621a1-openshift-default-dockercfg-b95rg\""
Sep 10 17:48:59.149324 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.149297    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"]
Sep 10 17:48:59.220488 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.220453    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-workload-certs\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.220488 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.220488    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-token\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.220720 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.220638    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-podinfo\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.220720 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.220670    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-credential-socket\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.220720 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.220705    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-envoy\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.220819 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.220734    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kpdq\" (UniqueName: \"kubernetes.io/projected/746856a7-c5c2-4381-82ad-43e7eacb4424-kube-api-access-6kpdq\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.220819 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.220813    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-workload-socket\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.220887 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.220832    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746856a7-c5c2-4381-82ad-43e7eacb4424-istiod-ca-cert\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.220887 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.220876    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-data\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.322026 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.321990    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-podinfo\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.322222 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.322031    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-credential-socket\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.322222 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.322071    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-envoy\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.322222 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.322094    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kpdq\" (UniqueName: \"kubernetes.io/projected/746856a7-c5c2-4381-82ad-43e7eacb4424-kube-api-access-6kpdq\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.322222 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.322160    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-workload-socket\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.322222 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.322187    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746856a7-c5c2-4381-82ad-43e7eacb4424-istiod-ca-cert\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.322526 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.322236    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-data\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.322526 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.322277    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-workload-certs\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.322526 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.322298    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-token\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.322701 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.322640    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-credential-socket\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.323019 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.322949    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-workload-socket\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.323019 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.322973    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-workload-certs\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.323184 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.323126    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-data\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.323591 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.323572    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746856a7-c5c2-4381-82ad-43e7eacb4424-istiod-ca-cert\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.324930 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.324908    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-podinfo\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.325022 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.324949    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-envoy\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.330904 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.330873    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-token\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.331156 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.331129    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kpdq\" (UniqueName: \"kubernetes.io/projected/746856a7-c5c2-4381-82ad-43e7eacb4424-kube-api-access-6kpdq\") pod \"e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") " pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.445165 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.445071    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:48:59.724349 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.724324    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"]
Sep 10 17:48:59.726388 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:48:59.726360    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746856a7_c5c2_4381_82ad_43e7eacb4424.slice/crio-c8c84e4c41b949d0aa7f5aab190b12602643981b51e48db7cf9e997d04e1e5ca WatchSource:0}: Error finding container c8c84e4c41b949d0aa7f5aab190b12602643981b51e48db7cf9e997d04e1e5ca: Status 404 returned error can't find the container with id c8c84e4c41b949d0aa7f5aab190b12602643981b51e48db7cf9e997d04e1e5ca
Sep 10 17:48:59.728754 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.728710    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:48:59.728839 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.728807    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:48:59.728880 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.728860    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:48:59.742541 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:48:59.742516    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn" event={"ID":"746856a7-c5c2-4381-82ad-43e7eacb4424","Type":"ContainerStarted","Data":"c8c84e4c41b949d0aa7f5aab190b12602643981b51e48db7cf9e997d04e1e5ca"}
Sep 10 17:49:00.685263 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:00.685230    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"]
Sep 10 17:49:00.685671 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:00.685443    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw" podUID="493420a1-4c9f-46dd-9819-9ad48049d271" containerName="istio-proxy" containerID="cri-o://103cfb12d7dfa7791d124f3c3f3d51fdafbefb9cb8e06afc9513362681966889" gracePeriod=30
Sep 10 17:49:00.749595 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:00.749553    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn" event={"ID":"746856a7-c5c2-4381-82ad-43e7eacb4424","Type":"ContainerStarted","Data":"c2d178ecff82ee4b433cd6b3dc24159d7b7c9a4b85b2da67d16ddc240bc78799"}
Sep 10 17:49:00.779000 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:00.778944    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn" podStartSLOduration=1.778928322 podStartE2EDuration="1.778928322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:49:00.778272259 +0000 UTC m=+1512.536799332" watchObservedRunningTime="2026-09-10 17:49:00.778928322 +0000 UTC m=+1512.537455392"
Sep 10 17:49:01.446230 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:01.446192    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:49:01.451016 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:01.450989    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:49:01.755051 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:01.754967    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:49:01.756074 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:01.756052    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:49:03.830783 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:03.830741    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 17:49:03.831249 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:03.830817    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw" podUID="493420a1-4c9f-46dd-9819-9ad48049d271" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 17:49:05.950782 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:05.950701    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:49:06.087901 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.087864    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493420a1-4c9f-46dd-9819-9ad48049d271-istiod-ca-cert\") pod \"493420a1-4c9f-46dd-9819-9ad48049d271\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") "
Sep 10 17:49:06.088085 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.087956    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-workload-socket\") pod \"493420a1-4c9f-46dd-9819-9ad48049d271\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") "
Sep 10 17:49:06.088085 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.088017    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-istio-envoy\") pod \"493420a1-4c9f-46dd-9819-9ad48049d271\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") "
Sep 10 17:49:06.088085 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.088052    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc5dz\" (UniqueName: \"kubernetes.io/projected/493420a1-4c9f-46dd-9819-9ad48049d271-kube-api-access-sc5dz\") pod \"493420a1-4c9f-46dd-9819-9ad48049d271\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") "
Sep 10 17:49:06.088248 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.088094    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493420a1-4c9f-46dd-9819-9ad48049d271-istio-podinfo\") pod \"493420a1-4c9f-46dd-9819-9ad48049d271\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") "
Sep 10 17:49:06.088248 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.088117    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-credential-socket\") pod \"493420a1-4c9f-46dd-9819-9ad48049d271\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") "
Sep 10 17:49:06.088248 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.088155    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-istio-data\") pod \"493420a1-4c9f-46dd-9819-9ad48049d271\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") "
Sep 10 17:49:06.088248 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.088186    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493420a1-4c9f-46dd-9819-9ad48049d271-istio-token\") pod \"493420a1-4c9f-46dd-9819-9ad48049d271\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") "
Sep 10 17:49:06.088248 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.088217    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-workload-certs\") pod \"493420a1-4c9f-46dd-9819-9ad48049d271\" (UID: \"493420a1-4c9f-46dd-9819-9ad48049d271\") "
Sep 10 17:49:06.088530 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.088275    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "493420a1-4c9f-46dd-9819-9ad48049d271" (UID: "493420a1-4c9f-46dd-9819-9ad48049d271"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:49:06.088530 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.088286    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493420a1-4c9f-46dd-9819-9ad48049d271-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "493420a1-4c9f-46dd-9819-9ad48049d271" (UID: "493420a1-4c9f-46dd-9819-9ad48049d271"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:49:06.088530 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.088469    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "493420a1-4c9f-46dd-9819-9ad48049d271" (UID: "493420a1-4c9f-46dd-9819-9ad48049d271"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:49:06.088693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.088621    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-credential-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.088693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.088623    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-istio-data" (OuterVolumeSpecName: "istio-data") pod "493420a1-4c9f-46dd-9819-9ad48049d271" (UID: "493420a1-4c9f-46dd-9819-9ad48049d271"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:49:06.088693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.088641    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493420a1-4c9f-46dd-9819-9ad48049d271-istiod-ca-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.088693 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.088657    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-workload-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.088894 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.088694    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "493420a1-4c9f-46dd-9819-9ad48049d271" (UID: "493420a1-4c9f-46dd-9819-9ad48049d271"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:49:06.090326 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.090302    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "493420a1-4c9f-46dd-9819-9ad48049d271" (UID: "493420a1-4c9f-46dd-9819-9ad48049d271"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:49:06.090326 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.090317    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493420a1-4c9f-46dd-9819-9ad48049d271-istio-token" (OuterVolumeSpecName: "istio-token") pod "493420a1-4c9f-46dd-9819-9ad48049d271" (UID: "493420a1-4c9f-46dd-9819-9ad48049d271"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:49:06.090746 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.090724    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/493420a1-4c9f-46dd-9819-9ad48049d271-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "493420a1-4c9f-46dd-9819-9ad48049d271" (UID: "493420a1-4c9f-46dd-9819-9ad48049d271"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:49:06.090805 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.090744    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493420a1-4c9f-46dd-9819-9ad48049d271-kube-api-access-sc5dz" (OuterVolumeSpecName: "kube-api-access-sc5dz") pod "493420a1-4c9f-46dd-9819-9ad48049d271" (UID: "493420a1-4c9f-46dd-9819-9ad48049d271"). InnerVolumeSpecName "kube-api-access-sc5dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:49:06.189634 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.189586    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-istio-envoy\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.189634 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.189625    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc5dz\" (UniqueName: \"kubernetes.io/projected/493420a1-4c9f-46dd-9819-9ad48049d271-kube-api-access-sc5dz\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.189634 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.189636    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493420a1-4c9f-46dd-9819-9ad48049d271-istio-podinfo\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.189634 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.189645    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-istio-data\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.189911 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.189653    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493420a1-4c9f-46dd-9819-9ad48049d271-istio-token\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.189911 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.189660    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493420a1-4c9f-46dd-9819-9ad48049d271-workload-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:49:06.784934 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.784899    2552 generic.go:356] "Generic (PLEG): container finished" podID="493420a1-4c9f-46dd-9819-9ad48049d271" containerID="103cfb12d7dfa7791d124f3c3f3d51fdafbefb9cb8e06afc9513362681966889" exitCode=0
Sep 10 17:49:06.785103 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.784946    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"
Sep 10 17:49:06.785103 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.784977    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw" event={"ID":"493420a1-4c9f-46dd-9819-9ad48049d271","Type":"ContainerDied","Data":"103cfb12d7dfa7791d124f3c3f3d51fdafbefb9cb8e06afc9513362681966889"}
Sep 10 17:49:06.785103 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.785038    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw" event={"ID":"493420a1-4c9f-46dd-9819-9ad48049d271","Type":"ContainerDied","Data":"0b0a8a768705327e184aeb007515d929cbc48aba39b419fc474f8b35cc805b69"}
Sep 10 17:49:06.785103 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.785061    2552 scope.go:117] "RemoveContainer" containerID="103cfb12d7dfa7791d124f3c3f3d51fdafbefb9cb8e06afc9513362681966889"
Sep 10 17:49:06.794942 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.794918    2552 scope.go:117] "RemoveContainer" containerID="929a03862aec8a765099779195bc7fbc3a336fa405d3cd8429c0900d2a2f385c"
Sep 10 17:49:06.804821 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.804624    2552 scope.go:117] "RemoveContainer" containerID="103cfb12d7dfa7791d124f3c3f3d51fdafbefb9cb8e06afc9513362681966889"
Sep 10 17:49:06.804957 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:49:06.804934    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"103cfb12d7dfa7791d124f3c3f3d51fdafbefb9cb8e06afc9513362681966889\": container with ID starting with 103cfb12d7dfa7791d124f3c3f3d51fdafbefb9cb8e06afc9513362681966889 not found: ID does not exist" containerID="103cfb12d7dfa7791d124f3c3f3d51fdafbefb9cb8e06afc9513362681966889"
Sep 10 17:49:06.805001 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.804965    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"103cfb12d7dfa7791d124f3c3f3d51fdafbefb9cb8e06afc9513362681966889"} err="failed to get container status \"103cfb12d7dfa7791d124f3c3f3d51fdafbefb9cb8e06afc9513362681966889\": rpc error: code = NotFound desc = could not find container \"103cfb12d7dfa7791d124f3c3f3d51fdafbefb9cb8e06afc9513362681966889\": container with ID starting with 103cfb12d7dfa7791d124f3c3f3d51fdafbefb9cb8e06afc9513362681966889 not found: ID does not exist"
Sep 10 17:49:06.805001 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.804984    2552 scope.go:117] "RemoveContainer" containerID="929a03862aec8a765099779195bc7fbc3a336fa405d3cd8429c0900d2a2f385c"
Sep 10 17:49:06.805273 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:49:06.805248    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929a03862aec8a765099779195bc7fbc3a336fa405d3cd8429c0900d2a2f385c\": container with ID starting with 929a03862aec8a765099779195bc7fbc3a336fa405d3cd8429c0900d2a2f385c not found: ID does not exist" containerID="929a03862aec8a765099779195bc7fbc3a336fa405d3cd8429c0900d2a2f385c"
Sep 10 17:49:06.805400 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.805277    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929a03862aec8a765099779195bc7fbc3a336fa405d3cd8429c0900d2a2f385c"} err="failed to get container status \"929a03862aec8a765099779195bc7fbc3a336fa405d3cd8429c0900d2a2f385c\": rpc error: code = NotFound desc = could not find container \"929a03862aec8a765099779195bc7fbc3a336fa405d3cd8429c0900d2a2f385c\": container with ID starting with 929a03862aec8a765099779195bc7fbc3a336fa405d3cd8429c0900d2a2f385c not found: ID does not exist"
Sep 10 17:49:06.808146 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.808122    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"]
Sep 10 17:49:06.812008 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:06.811980    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-762738-openshift-default-699567cb6-rp2jw"]
Sep 10 17:49:08.803465 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:08.803433    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="493420a1-4c9f-46dd-9819-9ad48049d271" path="/var/lib/kubelet/pods/493420a1-4c9f-46dd-9819-9ad48049d271/volumes"
Sep 10 17:49:10.707455 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:10.707417    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:49:20.861966 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:20.861930    2552 generic.go:356] "Generic (PLEG): container finished" podID="746856a7-c5c2-4381-82ad-43e7eacb4424" containerID="c2d178ecff82ee4b433cd6b3dc24159d7b7c9a4b85b2da67d16ddc240bc78799" exitCode=0
Sep 10 17:49:20.862401 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:20.861987    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn" event={"ID":"746856a7-c5c2-4381-82ad-43e7eacb4424","Type":"ContainerDied","Data":"c2d178ecff82ee4b433cd6b3dc24159d7b7c9a4b85b2da67d16ddc240bc78799"}
Sep 10 17:49:20.862401 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:20.862376    2552 scope.go:117] "RemoveContainer" containerID="c2d178ecff82ee4b433cd6b3dc24159d7b7c9a4b85b2da67d16ddc240bc78799"
Sep 10 17:49:20.863484 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:20.863450    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:49:20.863608 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:20.863566    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:49:20.863672 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:20.863618    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:49:21.873651 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:21.873613    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn" event={"ID":"746856a7-c5c2-4381-82ad-43e7eacb4424","Type":"ContainerStarted","Data":"d255eee6dd88ff32116b546ae82b3a4db8e2b1ab565b36791dc1205f4b5f75b0"}
Sep 10 17:49:22.446485 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:22.446431    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:49:22.446748 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:22.446595    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 10 17:49:22.446748 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:22.446645    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn" podUID="746856a7-c5c2-4381-82ad-43e7eacb4424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 10 17:49:23.445716 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:23.445678    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 10 17:49:23.446324 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:23.446290    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn" podUID="746856a7-c5c2-4381-82ad-43e7eacb4424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 10 17:49:24.446486 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:24.446434    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 10 17:49:24.447031 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:24.446530    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn" podUID="746856a7-c5c2-4381-82ad-43e7eacb4424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 10 17:49:25.445880 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:25.445836    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 10 17:49:25.446100 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:25.445913    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn" podUID="746856a7-c5c2-4381-82ad-43e7eacb4424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 10 17:49:26.446148 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:26.446104    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 10 17:49:26.446800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:26.446765    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn" podUID="746856a7-c5c2-4381-82ad-43e7eacb4424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 10 17:49:27.449852 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:27.449819    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:49:27.450217 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:27.450087    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:49:27.450816 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:27.450798    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:49:42.611319 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.611281    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"]
Sep 10 17:49:42.611710 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.611669    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493420a1-4c9f-46dd-9819-9ad48049d271" containerName="istio-proxy"
Sep 10 17:49:42.611710 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.611683    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="493420a1-4c9f-46dd-9819-9ad48049d271" containerName="istio-proxy"
Sep 10 17:49:42.611710 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.611702    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493420a1-4c9f-46dd-9819-9ad48049d271" containerName="istio-proxy"
Sep 10 17:49:42.611710 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.611707    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="493420a1-4c9f-46dd-9819-9ad48049d271" containerName="istio-proxy"
Sep 10 17:49:42.611876 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.611772    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="493420a1-4c9f-46dd-9819-9ad48049d271" containerName="istio-proxy"
Sep 10 17:49:42.613885 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.613869    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.616781 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.616753    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-dc47cd\"/\"openshift-service-ca.crt\""
Sep 10 17:49:42.617989 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.617966    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-dc47cd\"/\"default-dockercfg-v5fdj\""
Sep 10 17:49:42.618141 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.618122    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-dc47cd\"/\"ipp-route-dc47cd-kserve-self-signed-certs\""
Sep 10 17:49:42.618205 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.618190    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-dc47cd\"/\"kube-root-ca.crt\""
Sep 10 17:49:42.623860 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.623835    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"]
Sep 10 17:49:42.751376 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.751338    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-tmp-dir\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.751577 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.751391    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a9789b-07b4-4510-86db-4905eb29d852-tls-certs\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.751577 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.751485    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-home\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.751661 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.751593    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7frsz\" (UniqueName: \"kubernetes.io/projected/90a9789b-07b4-4510-86db-4905eb29d852-kube-api-access-7frsz\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.751661 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.751636    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-dshm\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.751732 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.751662    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-model-cache\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.852675 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.852639    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a9789b-07b4-4510-86db-4905eb29d852-tls-certs\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.852873 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.852701    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-home\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.852873 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.852737    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7frsz\" (UniqueName: \"kubernetes.io/projected/90a9789b-07b4-4510-86db-4905eb29d852-kube-api-access-7frsz\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.852873 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.852762    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-dshm\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.853066 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.852908    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-model-cache\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.853066 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.852991    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-tmp-dir\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.853190 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.853167    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-home\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.853341 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.853306    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-tmp-dir\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.853341 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.853333    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-model-cache\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.855043 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.855026    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-dshm\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.855214 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.855196    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a9789b-07b4-4510-86db-4905eb29d852-tls-certs\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.864106 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.864052    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7frsz\" (UniqueName: \"kubernetes.io/projected/90a9789b-07b4-4510-86db-4905eb29d852-kube-api-access-7frsz\") pod \"ipp-route-dc47cd-kserve-5d9b9445d7-xf89b\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") " pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:42.924984 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:42.924943    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:43.204554 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:43.204520    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"]
Sep 10 17:49:43.206764 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:49:43.206739    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90a9789b_07b4_4510_86db_4905eb29d852.slice/crio-37b4db9ee0d874a33539e70d32abc67138b8adb035057ab672a75b8b216153da WatchSource:0}: Error finding container 37b4db9ee0d874a33539e70d32abc67138b8adb035057ab672a75b8b216153da: Status 404 returned error can't find the container with id 37b4db9ee0d874a33539e70d32abc67138b8adb035057ab672a75b8b216153da
Sep 10 17:49:43.987848 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:43.987811    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b" event={"ID":"90a9789b-07b4-4510-86db-4905eb29d852","Type":"ContainerStarted","Data":"6e608f8d0c47d1621588515ef0a930155785f59fd26e4e25dfcfebc998090c12"}
Sep 10 17:49:43.987848 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:43.987858    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b" event={"ID":"90a9789b-07b4-4510-86db-4905eb29d852","Type":"ContainerStarted","Data":"37b4db9ee0d874a33539e70d32abc67138b8adb035057ab672a75b8b216153da"}
Sep 10 17:49:44.009440 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:44.009386    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b" podStartSLOduration=2.009371043 podStartE2EDuration="2.009371043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:49:44.006268959 +0000 UTC m=+1555.764796030" watchObservedRunningTime="2026-09-10 17:49:44.009371043 +0000 UTC m=+1555.767898136"
Sep 10 17:49:45.601188 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:45.601144    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:49:52.925653 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:52.925614    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:52.926080 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:52.925667    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:52.938709 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:52.938672    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:53.048764 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:53.048731    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:49:58.600047 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:49:58.600007    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:50:06.486865 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.486832    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"]
Sep 10 17:50:06.487331 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.487278    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="493420a1-4c9f-46dd-9819-9ad48049d271" containerName="istio-proxy"
Sep 10 17:50:06.490852 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.490832    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.493800 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.493775    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-8e4477-openshift-default-dockercfg-nrdx8\""
Sep 10 17:50:06.503339 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.503307    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"]
Sep 10 17:50:06.589311 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.589267    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f01190d-3054-4beb-a025-b0a555ac105d-istio-token\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.589538 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.589331    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-workload-socket\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.589538 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.589356    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55j4c\" (UniqueName: \"kubernetes.io/projected/6f01190d-3054-4beb-a025-b0a555ac105d-kube-api-access-55j4c\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.589538 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.589419    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-credential-socket\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.589538 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.589442    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-workload-certs\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.589538 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.589470    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f01190d-3054-4beb-a025-b0a555ac105d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.589538 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.589511    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f01190d-3054-4beb-a025-b0a555ac105d-istio-podinfo\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.589819 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.589593    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-istio-data\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.589819 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.589634    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-istio-envoy\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.690993 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.690954    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-istio-data\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.691168 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.691003    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-istio-envoy\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.691168 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.691052    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f01190d-3054-4beb-a025-b0a555ac105d-istio-token\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.691168 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.691092    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-workload-socket\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.691168 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.691117    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55j4c\" (UniqueName: \"kubernetes.io/projected/6f01190d-3054-4beb-a025-b0a555ac105d-kube-api-access-55j4c\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.691380 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.691224    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-credential-socket\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.691380 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.691250    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-workload-certs\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.691380 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.691282    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f01190d-3054-4beb-a025-b0a555ac105d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.691380 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.691312    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f01190d-3054-4beb-a025-b0a555ac105d-istio-podinfo\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.691631 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.691454    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-workload-socket\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.691631 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.691578    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-istio-data\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.691777 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.691736    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-workload-certs\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.691846 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.691776    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-credential-socket\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.692096 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.692062    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f01190d-3054-4beb-a025-b0a555ac105d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.693557 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.693528    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-istio-envoy\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.693909 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.693775    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f01190d-3054-4beb-a025-b0a555ac105d-istio-podinfo\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.699583 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.699482    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f01190d-3054-4beb-a025-b0a555ac105d-istio-token\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.699971 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.699952    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55j4c\" (UniqueName: \"kubernetes.io/projected/6f01190d-3054-4beb-a025-b0a555ac105d-kube-api-access-55j4c\") pod \"e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") " pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:06.801951 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:06.801820    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:07.077144 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:07.077107    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"]
Sep 10 17:50:07.078608 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:50:07.078582    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f01190d_3054_4beb_a025_b0a555ac105d.slice/crio-fe5bb2035628e1255b8c53763f403d8fb725b01c6cf763a4db0c64c999f6a1f1 WatchSource:0}: Error finding container fe5bb2035628e1255b8c53763f403d8fb725b01c6cf763a4db0c64c999f6a1f1: Status 404 returned error can't find the container with id fe5bb2035628e1255b8c53763f403d8fb725b01c6cf763a4db0c64c999f6a1f1
Sep 10 17:50:07.081090 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:07.081048    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:50:07.081172 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:07.081149    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:50:07.081212 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:07.081196    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:50:07.116978 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:07.116948    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4" event={"ID":"6f01190d-3054-4beb-a025-b0a555ac105d","Type":"ContainerStarted","Data":"fe5bb2035628e1255b8c53763f403d8fb725b01c6cf763a4db0c64c999f6a1f1"}
Sep 10 17:50:08.131717 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:08.131681    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4" event={"ID":"6f01190d-3054-4beb-a025-b0a555ac105d","Type":"ContainerStarted","Data":"86f8c90e1be92fa782ff1f2470aab14909473ae202471b08e0d4fe5175b9b01f"}
Sep 10 17:50:08.153308 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:08.153254    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4" podStartSLOduration=2.153237398 podStartE2EDuration="2.153237398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:50:08.151760995 +0000 UTC m=+1579.910288096" watchObservedRunningTime="2026-09-10 17:50:08.153237398 +0000 UTC m=+1579.911764467"
Sep 10 17:50:08.803143 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:08.803106    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:08.807482 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:08.807454    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:09.137560 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:09.137531    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:09.138438 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:09.138417    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:18.115476 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:18.115417    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:50:28.248663 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:28.248626    2552 generic.go:356] "Generic (PLEG): container finished" podID="6f01190d-3054-4beb-a025-b0a555ac105d" containerID="86f8c90e1be92fa782ff1f2470aab14909473ae202471b08e0d4fe5175b9b01f" exitCode=0
Sep 10 17:50:28.249072 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:28.248696    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4" event={"ID":"6f01190d-3054-4beb-a025-b0a555ac105d","Type":"ContainerDied","Data":"86f8c90e1be92fa782ff1f2470aab14909473ae202471b08e0d4fe5175b9b01f"}
Sep 10 17:50:28.249167 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:28.249148    2552 scope.go:117] "RemoveContainer" containerID="86f8c90e1be92fa782ff1f2470aab14909473ae202471b08e0d4fe5175b9b01f"
Sep 10 17:50:28.250569 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:28.250537    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:50:28.250645 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:28.250627    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:50:28.250687 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:28.250677    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:50:29.260864 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:29.260817    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4" event={"ID":"6f01190d-3054-4beb-a025-b0a555ac105d","Type":"ContainerStarted","Data":"3ad8a013a83d4cc65649ae1ab4ffa2708959b69f822ea202515d4de58ff61b52"}
Sep 10 17:50:29.802307 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:29.802268    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:29.802582 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:29.802557    2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 10 17:50:29.802639 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:29.802616    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4" podUID="6f01190d-3054-4beb-a025-b0a555ac105d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 10 17:50:30.802567 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:30.802532    2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 10 17:50:30.802954 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:30.802587    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4" podUID="6f01190d-3054-4beb-a025-b0a555ac105d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 10 17:50:31.802345 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:31.802309    2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 10 17:50:31.802557 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:31.802365    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4" podUID="6f01190d-3054-4beb-a025-b0a555ac105d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 10 17:50:32.803036 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:32.802998    2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 10 17:50:32.803435 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:32.803066    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4" podUID="6f01190d-3054-4beb-a025-b0a555ac105d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 10 17:50:33.802573 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:33.802526    2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 10 17:50:33.802766 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:33.802590    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4" podUID="6f01190d-3054-4beb-a025-b0a555ac105d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 10 17:50:34.806572 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:34.806539    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:34.807005 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:34.806803    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:34.807602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:34.807585    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:35.798202 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:35.798165    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx"]
Sep 10 17:50:35.801117 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:35.801096    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx"
Sep 10 17:50:35.807128 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:35.807099    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 17:50:35.807128 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:35.807102    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:50:35.817377 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:35.817344    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx"]
Sep 10 17:50:35.899323 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:35.899284    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc41ce6-ec03-466c-9643-91c75105785f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx\" (UID: \"ccc41ce6-ec03-466c-9643-91c75105785f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx"
Sep 10 17:50:36.000222 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:36.000178    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc41ce6-ec03-466c-9643-91c75105785f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx\" (UID: \"ccc41ce6-ec03-466c-9643-91c75105785f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx"
Sep 10 17:50:36.000875 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:36.000852    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc41ce6-ec03-466c-9643-91c75105785f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx\" (UID: \"ccc41ce6-ec03-466c-9643-91c75105785f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx"
Sep 10 17:50:36.111594 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:36.111553    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx"
Sep 10 17:50:36.385088 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:36.385058    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx"]
Sep 10 17:50:36.387274 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:50:36.387245    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc41ce6_ec03_466c_9643_91c75105785f.slice/crio-83fc7df08e27f7b5824f2cc451bec1b22adbf2e22daa642b428e3462a2b8e37e WatchSource:0}: Error finding container 83fc7df08e27f7b5824f2cc451bec1b22adbf2e22daa642b428e3462a2b8e37e: Status 404 returned error can't find the container with id 83fc7df08e27f7b5824f2cc451bec1b22adbf2e22daa642b428e3462a2b8e37e
Sep 10 17:50:37.312222 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:37.312122    2552 generic.go:356] "Generic (PLEG): container finished" podID="ccc41ce6-ec03-466c-9643-91c75105785f" containerID="bc13cbefab065212042cd226dafc8679cd91b32575ecfc25cecdcfc16634b02e" exitCode=0
Sep 10 17:50:37.312611 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:37.312209    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx" event={"ID":"ccc41ce6-ec03-466c-9643-91c75105785f","Type":"ContainerDied","Data":"bc13cbefab065212042cd226dafc8679cd91b32575ecfc25cecdcfc16634b02e"}
Sep 10 17:50:37.312611 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:37.312289    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx" event={"ID":"ccc41ce6-ec03-466c-9643-91c75105785f","Type":"ContainerStarted","Data":"83fc7df08e27f7b5824f2cc451bec1b22adbf2e22daa642b428e3462a2b8e37e"}
Sep 10 17:50:38.456534 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:38.456487    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx"
Sep 10 17:50:38.627543 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:38.627469    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc41ce6-ec03-466c-9643-91c75105785f-maas-api-service-ca\") pod \"ccc41ce6-ec03-466c-9643-91c75105785f\" (UID: \"ccc41ce6-ec03-466c-9643-91c75105785f\") "
Sep 10 17:50:38.627877 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:38.627853    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc41ce6-ec03-466c-9643-91c75105785f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ccc41ce6-ec03-466c-9643-91c75105785f" (UID: "ccc41ce6-ec03-466c-9643-91c75105785f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:50:38.728416 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:38.728377    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc41ce6-ec03-466c-9643-91c75105785f-maas-api-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:50:39.324942 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:39.324913    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx"
Sep 10 17:50:39.325128 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:39.324952    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx" event={"ID":"ccc41ce6-ec03-466c-9643-91c75105785f","Type":"ContainerDied","Data":"83fc7df08e27f7b5824f2cc451bec1b22adbf2e22daa642b428e3462a2b8e37e"}
Sep 10 17:50:39.325128 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:39.325006    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83fc7df08e27f7b5824f2cc451bec1b22adbf2e22daa642b428e3462a2b8e37e"
Sep 10 17:50:41.959090 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:41.959000    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx"]
Sep 10 17:50:41.962308 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:41.962262    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e4477-kpblx"]
Sep 10 17:50:42.804454 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:42.804410    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccc41ce6-ec03-466c-9643-91c75105785f" path="/var/lib/kubelet/pods/ccc41ce6-ec03-466c-9643-91c75105785f/volumes"
Sep 10 17:50:46.095590 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:46.095547    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:50:48.784402 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:48.784364    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"]
Sep 10 17:50:48.784863 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:48.784661    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4" podUID="6f01190d-3054-4beb-a025-b0a555ac105d" containerName="istio-proxy" containerID="cri-o://3ad8a013a83d4cc65649ae1ab4ffa2708959b69f822ea202515d4de58ff61b52" gracePeriod=30
Sep 10 17:50:49.020252 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:49.020221    2552 scope.go:117] "RemoveContainer" containerID="377a6f0ab733eb22ec91f87fa686e5b57daaf716834e68f15edb57456ee5c9ea"
Sep 10 17:50:49.809884 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:49.809842    2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 17:50:49.810273 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:49.809909    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4" podUID="6f01190d-3054-4beb-a025-b0a555ac105d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 17:50:52.327265 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:52.327225    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt"]
Sep 10 17:50:52.327678 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:52.327664    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccc41ce6-ec03-466c-9643-91c75105785f" containerName="revoke-keys"
Sep 10 17:50:52.327723 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:52.327679    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccc41ce6-ec03-466c-9643-91c75105785f" containerName="revoke-keys"
Sep 10 17:50:52.327758 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:52.327749    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccc41ce6-ec03-466c-9643-91c75105785f" containerName="revoke-keys"
Sep 10 17:50:52.329683 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:52.329665    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt"
Sep 10 17:50:52.332828 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:52.332804    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:50:52.332993 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:52.332975    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 17:50:52.340347 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:52.340309    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt"]
Sep 10 17:50:52.367727 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:52.367688    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d0acb4-a86e-4db0-8977-8ac64a52f5e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt\" (UID: \"f4d0acb4-a86e-4db0-8977-8ac64a52f5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt"
Sep 10 17:50:52.468841 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:52.468797    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d0acb4-a86e-4db0-8977-8ac64a52f5e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt\" (UID: \"f4d0acb4-a86e-4db0-8977-8ac64a52f5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt"
Sep 10 17:50:52.469551 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:52.469489    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d0acb4-a86e-4db0-8977-8ac64a52f5e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt\" (UID: \"f4d0acb4-a86e-4db0-8977-8ac64a52f5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt"
Sep 10 17:50:52.641779 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:52.641737    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt"
Sep 10 17:50:52.917903 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:52.917873    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt"]
Sep 10 17:50:52.919991 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:50:52.919959    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d0acb4_a86e_4db0_8977_8ac64a52f5e1.slice/crio-8716ff90ff3598acee0946bde4d737534118d4e5c25e4e5bde1dcae1b4d5c4d9 WatchSource:0}: Error finding container 8716ff90ff3598acee0946bde4d737534118d4e5c25e4e5bde1dcae1b4d5c4d9: Status 404 returned error can't find the container with id 8716ff90ff3598acee0946bde4d737534118d4e5c25e4e5bde1dcae1b4d5c4d9
Sep 10 17:50:53.402724 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:53.402684    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt" event={"ID":"f4d0acb4-a86e-4db0-8977-8ac64a52f5e1","Type":"ContainerStarted","Data":"4736e5cc4c7ffe2a0768f3f661c85bf84fb47bfae04b7d045570c2a9cac9684d"}
Sep 10 17:50:53.403105 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:53.402749    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt" event={"ID":"f4d0acb4-a86e-4db0-8977-8ac64a52f5e1","Type":"ContainerStarted","Data":"8716ff90ff3598acee0946bde4d737534118d4e5c25e4e5bde1dcae1b4d5c4d9"}
Sep 10 17:50:53.419971 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:53.419916    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt" podStartSLOduration=1.419901153 podStartE2EDuration="1.419901153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:50:53.419283076 +0000 UTC m=+1625.177810147" watchObservedRunningTime="2026-09-10 17:50:53.419901153 +0000 UTC m=+1625.178428223"
Sep 10 17:50:54.036385 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.036357    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:54.085102 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.085066    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-istio-envoy\") pod \"6f01190d-3054-4beb-a025-b0a555ac105d\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") "
Sep 10 17:50:54.085102 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.085105    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-istio-data\") pod \"6f01190d-3054-4beb-a025-b0a555ac105d\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") "
Sep 10 17:50:54.085337 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.085144    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-credential-socket\") pod \"6f01190d-3054-4beb-a025-b0a555ac105d\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") "
Sep 10 17:50:54.085337 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.085189    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55j4c\" (UniqueName: \"kubernetes.io/projected/6f01190d-3054-4beb-a025-b0a555ac105d-kube-api-access-55j4c\") pod \"6f01190d-3054-4beb-a025-b0a555ac105d\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") "
Sep 10 17:50:54.085337 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.085268    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-workload-socket\") pod \"6f01190d-3054-4beb-a025-b0a555ac105d\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") "
Sep 10 17:50:54.085337 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.085290    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-workload-certs\") pod \"6f01190d-3054-4beb-a025-b0a555ac105d\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") "
Sep 10 17:50:54.085337 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.085315    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f01190d-3054-4beb-a025-b0a555ac105d-istio-podinfo\") pod \"6f01190d-3054-4beb-a025-b0a555ac105d\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") "
Sep 10 17:50:54.085616 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.085403    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f01190d-3054-4beb-a025-b0a555ac105d-istio-token\") pod \"6f01190d-3054-4beb-a025-b0a555ac105d\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") "
Sep 10 17:50:54.085616 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.085425    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f01190d-3054-4beb-a025-b0a555ac105d" (UID: "6f01190d-3054-4beb-a025-b0a555ac105d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:50:54.085616 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.085438    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f01190d-3054-4beb-a025-b0a555ac105d-istiod-ca-cert\") pod \"6f01190d-3054-4beb-a025-b0a555ac105d\" (UID: \"6f01190d-3054-4beb-a025-b0a555ac105d\") "
Sep 10 17:50:54.085616 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.085539    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f01190d-3054-4beb-a025-b0a555ac105d" (UID: "6f01190d-3054-4beb-a025-b0a555ac105d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:50:54.085832 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.085798    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f01190d-3054-4beb-a025-b0a555ac105d" (UID: "6f01190d-3054-4beb-a025-b0a555ac105d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:50:54.085832 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.085734    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f01190d-3054-4beb-a025-b0a555ac105d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f01190d-3054-4beb-a025-b0a555ac105d" (UID: "6f01190d-3054-4beb-a025-b0a555ac105d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:50:54.085965 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.085920    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-workload-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:50:54.086111 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.085997    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f01190d-3054-4beb-a025-b0a555ac105d-istiod-ca-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:50:54.086111 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.086015    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-istio-data\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:50:54.086111 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.086015    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f01190d-3054-4beb-a025-b0a555ac105d" (UID: "6f01190d-3054-4beb-a025-b0a555ac105d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:50:54.086111 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.086027    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-credential-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:50:54.087815 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.087786    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f01190d-3054-4beb-a025-b0a555ac105d-kube-api-access-55j4c" (OuterVolumeSpecName: "kube-api-access-55j4c") pod "6f01190d-3054-4beb-a025-b0a555ac105d" (UID: "6f01190d-3054-4beb-a025-b0a555ac105d"). InnerVolumeSpecName "kube-api-access-55j4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:50:54.087977 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.087955    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f01190d-3054-4beb-a025-b0a555ac105d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f01190d-3054-4beb-a025-b0a555ac105d" (UID: "6f01190d-3054-4beb-a025-b0a555ac105d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:50:54.088056 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.087954    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f01190d-3054-4beb-a025-b0a555ac105d" (UID: "6f01190d-3054-4beb-a025-b0a555ac105d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:50:54.088114 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.088086    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f01190d-3054-4beb-a025-b0a555ac105d-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f01190d-3054-4beb-a025-b0a555ac105d" (UID: "6f01190d-3054-4beb-a025-b0a555ac105d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:50:54.186599 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.186558    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55j4c\" (UniqueName: \"kubernetes.io/projected/6f01190d-3054-4beb-a025-b0a555ac105d-kube-api-access-55j4c\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:50:54.186599 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.186591    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-workload-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:50:54.186599 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.186600    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f01190d-3054-4beb-a025-b0a555ac105d-istio-podinfo\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:50:54.186599 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.186608    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f01190d-3054-4beb-a025-b0a555ac105d-istio-token\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:50:54.186863 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.186616    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f01190d-3054-4beb-a025-b0a555ac105d-istio-envoy\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:50:54.411012 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.410975    2552 generic.go:356] "Generic (PLEG): container finished" podID="f4d0acb4-a86e-4db0-8977-8ac64a52f5e1" containerID="4736e5cc4c7ffe2a0768f3f661c85bf84fb47bfae04b7d045570c2a9cac9684d" exitCode=0
Sep 10 17:50:54.411477 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.411059    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt" event={"ID":"f4d0acb4-a86e-4db0-8977-8ac64a52f5e1","Type":"ContainerDied","Data":"4736e5cc4c7ffe2a0768f3f661c85bf84fb47bfae04b7d045570c2a9cac9684d"}
Sep 10 17:50:54.413421 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.413390    2552 generic.go:356] "Generic (PLEG): container finished" podID="6f01190d-3054-4beb-a025-b0a555ac105d" containerID="3ad8a013a83d4cc65649ae1ab4ffa2708959b69f822ea202515d4de58ff61b52" exitCode=0
Sep 10 17:50:54.413536 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.413433    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"
Sep 10 17:50:54.413536 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.413456    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4" event={"ID":"6f01190d-3054-4beb-a025-b0a555ac105d","Type":"ContainerDied","Data":"3ad8a013a83d4cc65649ae1ab4ffa2708959b69f822ea202515d4de58ff61b52"}
Sep 10 17:50:54.413536 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.413529    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4" event={"ID":"6f01190d-3054-4beb-a025-b0a555ac105d","Type":"ContainerDied","Data":"fe5bb2035628e1255b8c53763f403d8fb725b01c6cf763a4db0c64c999f6a1f1"}
Sep 10 17:50:54.413660 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.413560    2552 scope.go:117] "RemoveContainer" containerID="3ad8a013a83d4cc65649ae1ab4ffa2708959b69f822ea202515d4de58ff61b52"
Sep 10 17:50:54.424565 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.424541    2552 scope.go:117] "RemoveContainer" containerID="86f8c90e1be92fa782ff1f2470aab14909473ae202471b08e0d4fe5175b9b01f"
Sep 10 17:50:54.435587 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.435563    2552 scope.go:117] "RemoveContainer" containerID="3ad8a013a83d4cc65649ae1ab4ffa2708959b69f822ea202515d4de58ff61b52"
Sep 10 17:50:54.436034 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:50:54.436006    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad8a013a83d4cc65649ae1ab4ffa2708959b69f822ea202515d4de58ff61b52\": container with ID starting with 3ad8a013a83d4cc65649ae1ab4ffa2708959b69f822ea202515d4de58ff61b52 not found: ID does not exist" containerID="3ad8a013a83d4cc65649ae1ab4ffa2708959b69f822ea202515d4de58ff61b52"
Sep 10 17:50:54.436124 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.436042    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad8a013a83d4cc65649ae1ab4ffa2708959b69f822ea202515d4de58ff61b52"} err="failed to get container status \"3ad8a013a83d4cc65649ae1ab4ffa2708959b69f822ea202515d4de58ff61b52\": rpc error: code = NotFound desc = could not find container \"3ad8a013a83d4cc65649ae1ab4ffa2708959b69f822ea202515d4de58ff61b52\": container with ID starting with 3ad8a013a83d4cc65649ae1ab4ffa2708959b69f822ea202515d4de58ff61b52 not found: ID does not exist"
Sep 10 17:50:54.436124 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.436062    2552 scope.go:117] "RemoveContainer" containerID="86f8c90e1be92fa782ff1f2470aab14909473ae202471b08e0d4fe5175b9b01f"
Sep 10 17:50:54.436435 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:50:54.436413    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f8c90e1be92fa782ff1f2470aab14909473ae202471b08e0d4fe5175b9b01f\": container with ID starting with 86f8c90e1be92fa782ff1f2470aab14909473ae202471b08e0d4fe5175b9b01f not found: ID does not exist" containerID="86f8c90e1be92fa782ff1f2470aab14909473ae202471b08e0d4fe5175b9b01f"
Sep 10 17:50:54.436483 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.436445    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f8c90e1be92fa782ff1f2470aab14909473ae202471b08e0d4fe5175b9b01f"} err="failed to get container status \"86f8c90e1be92fa782ff1f2470aab14909473ae202471b08e0d4fe5175b9b01f\": rpc error: code = NotFound desc = could not find container \"86f8c90e1be92fa782ff1f2470aab14909473ae202471b08e0d4fe5175b9b01f\": container with ID starting with 86f8c90e1be92fa782ff1f2470aab14909473ae202471b08e0d4fe5175b9b01f not found: ID does not exist"
Sep 10 17:50:54.443707 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.443673    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"]
Sep 10 17:50:54.448136 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.448103    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e4477-openshift-default-5b95bcc846-xx4x4"]
Sep 10 17:50:54.804441 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:54.804351    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f01190d-3054-4beb-a025-b0a555ac105d" path="/var/lib/kubelet/pods/6f01190d-3054-4beb-a025-b0a555ac105d/volumes"
Sep 10 17:50:55.556261 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:55.556229    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt"
Sep 10 17:50:55.598864 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:55.598816    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d0acb4-a86e-4db0-8977-8ac64a52f5e1-maas-api-service-ca\") pod \"f4d0acb4-a86e-4db0-8977-8ac64a52f5e1\" (UID: \"f4d0acb4-a86e-4db0-8977-8ac64a52f5e1\") "
Sep 10 17:50:55.599174 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:55.599147    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d0acb4-a86e-4db0-8977-8ac64a52f5e1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f4d0acb4-a86e-4db0-8977-8ac64a52f5e1" (UID: "f4d0acb4-a86e-4db0-8977-8ac64a52f5e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:50:55.700550 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:55.700411    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d0acb4-a86e-4db0-8977-8ac64a52f5e1-maas-api-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:50:56.428470 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:56.428440    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt"
Sep 10 17:50:56.428752 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:56.428477    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt" event={"ID":"f4d0acb4-a86e-4db0-8977-8ac64a52f5e1","Type":"ContainerDied","Data":"8716ff90ff3598acee0946bde4d737534118d4e5c25e4e5bde1dcae1b4d5c4d9"}
Sep 10 17:50:56.428752 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:50:56.428555    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8716ff90ff3598acee0946bde4d737534118d4e5c25e4e5bde1dcae1b4d5c4d9"
Sep 10 17:51:06.008717 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:06.008680    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:51:08.093896 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:08.093858    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt"]
Sep 10 17:51:08.096703 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:08.096677    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dc47cd-26xmt"]
Sep 10 17:51:08.803721 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:08.803688    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4d0acb4-a86e-4db0-8977-8ac64a52f5e1" path="/var/lib/kubelet/pods/f4d0acb4-a86e-4db0-8977-8ac64a52f5e1/volumes"
Sep 10 17:51:13.442233 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:13.442179    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"]
Sep 10 17:51:18.396348 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:18.396309    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:51:28.466748 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.466689    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b" podUID="90a9789b-07b4-4510-86db-4905eb29d852" containerName="main" containerID="cri-o://6e608f8d0c47d1621588515ef0a930155785f59fd26e4e25dfcfebc998090c12" gracePeriod=45
Sep 10 17:51:28.604844 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.604808    2552 generic.go:356] "Generic (PLEG): container finished" podID="90a9789b-07b4-4510-86db-4905eb29d852" containerID="6e608f8d0c47d1621588515ef0a930155785f59fd26e4e25dfcfebc998090c12" exitCode=0
Sep 10 17:51:28.605020 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.604898    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b" event={"ID":"90a9789b-07b4-4510-86db-4905eb29d852","Type":"ContainerDied","Data":"6e608f8d0c47d1621588515ef0a930155785f59fd26e4e25dfcfebc998090c12"}
Sep 10 17:51:28.718403 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.718335    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:51:28.837623 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.837582    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a9789b-07b4-4510-86db-4905eb29d852-tls-certs\") pod \"90a9789b-07b4-4510-86db-4905eb29d852\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") "
Sep 10 17:51:28.837623 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.837629    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7frsz\" (UniqueName: \"kubernetes.io/projected/90a9789b-07b4-4510-86db-4905eb29d852-kube-api-access-7frsz\") pod \"90a9789b-07b4-4510-86db-4905eb29d852\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") "
Sep 10 17:51:28.837860 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.837680    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-dshm\") pod \"90a9789b-07b4-4510-86db-4905eb29d852\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") "
Sep 10 17:51:28.837860 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.837748    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-model-cache\") pod \"90a9789b-07b4-4510-86db-4905eb29d852\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") "
Sep 10 17:51:28.837860 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.837799    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-home\") pod \"90a9789b-07b4-4510-86db-4905eb29d852\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") "
Sep 10 17:51:28.837860 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.837851    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-tmp-dir\") pod \"90a9789b-07b4-4510-86db-4905eb29d852\" (UID: \"90a9789b-07b4-4510-86db-4905eb29d852\") "
Sep 10 17:51:28.838090 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.838063    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-model-cache" (OuterVolumeSpecName: "model-cache") pod "90a9789b-07b4-4510-86db-4905eb29d852" (UID: "90a9789b-07b4-4510-86db-4905eb29d852"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:51:28.838162 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.838105    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-home" (OuterVolumeSpecName: "home") pod "90a9789b-07b4-4510-86db-4905eb29d852" (UID: "90a9789b-07b4-4510-86db-4905eb29d852"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:51:28.838226 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.838193    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90a9789b-07b4-4510-86db-4905eb29d852" (UID: "90a9789b-07b4-4510-86db-4905eb29d852"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:51:28.838283 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.838239    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-home\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:51:28.838283 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.838259    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-model-cache\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:51:28.840038 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.840019    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-dshm" (OuterVolumeSpecName: "dshm") pod "90a9789b-07b4-4510-86db-4905eb29d852" (UID: "90a9789b-07b4-4510-86db-4905eb29d852"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:51:28.840107 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.840059    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a9789b-07b4-4510-86db-4905eb29d852-kube-api-access-7frsz" (OuterVolumeSpecName: "kube-api-access-7frsz") pod "90a9789b-07b4-4510-86db-4905eb29d852" (UID: "90a9789b-07b4-4510-86db-4905eb29d852"). InnerVolumeSpecName "kube-api-access-7frsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:51:28.840149 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.840114    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a9789b-07b4-4510-86db-4905eb29d852-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90a9789b-07b4-4510-86db-4905eb29d852" (UID: "90a9789b-07b4-4510-86db-4905eb29d852"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:51:28.938791 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.938751    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-tmp-dir\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:51:28.938791 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.938784    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a9789b-07b4-4510-86db-4905eb29d852-tls-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:51:28.938791 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.938796    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7frsz\" (UniqueName: \"kubernetes.io/projected/90a9789b-07b4-4510-86db-4905eb29d852-kube-api-access-7frsz\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:51:28.939018 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:28.938804    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a9789b-07b4-4510-86db-4905eb29d852-dshm\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:51:29.611891 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:29.611859    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"
Sep 10 17:51:29.611891 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:29.611873    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b" event={"ID":"90a9789b-07b4-4510-86db-4905eb29d852","Type":"ContainerDied","Data":"37b4db9ee0d874a33539e70d32abc67138b8adb035057ab672a75b8b216153da"}
Sep 10 17:51:29.612467 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:29.611924    2552 scope.go:117] "RemoveContainer" containerID="6e608f8d0c47d1621588515ef0a930155785f59fd26e4e25dfcfebc998090c12"
Sep 10 17:51:29.633394 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:29.633359    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"]
Sep 10 17:51:29.637163 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:29.637136    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-dc47cd/ipp-route-dc47cd-kserve-5d9b9445d7-xf89b"]
Sep 10 17:51:30.802932 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:51:30.802900    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90a9789b-07b4-4510-86db-4905eb29d852" path="/var/lib/kubelet/pods/90a9789b-07b4-4510-86db-4905eb29d852/volumes"
Sep 10 17:52:28.208011 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.207977    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm"]
Sep 10 17:52:28.208454 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.208355    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f01190d-3054-4beb-a025-b0a555ac105d" containerName="istio-proxy"
Sep 10 17:52:28.208454 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.208367    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f01190d-3054-4beb-a025-b0a555ac105d" containerName="istio-proxy"
Sep 10 17:52:28.208454 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.208381    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90a9789b-07b4-4510-86db-4905eb29d852" containerName="main"
Sep 10 17:52:28.208454 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.208387    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="90a9789b-07b4-4510-86db-4905eb29d852" containerName="main"
Sep 10 17:52:28.208454 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.208403    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4d0acb4-a86e-4db0-8977-8ac64a52f5e1" containerName="revoke-keys"
Sep 10 17:52:28.208454 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.208409    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4d0acb4-a86e-4db0-8977-8ac64a52f5e1" containerName="revoke-keys"
Sep 10 17:52:28.208692 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.208470    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4d0acb4-a86e-4db0-8977-8ac64a52f5e1" containerName="revoke-keys"
Sep 10 17:52:28.208692 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.208478    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f01190d-3054-4beb-a025-b0a555ac105d" containerName="istio-proxy"
Sep 10 17:52:28.208692 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.208485    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="90a9789b-07b4-4510-86db-4905eb29d852" containerName="main"
Sep 10 17:52:28.208692 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.208506    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f01190d-3054-4beb-a025-b0a555ac105d" containerName="istio-proxy"
Sep 10 17:52:28.211602 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.211578    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm"
Sep 10 17:52:28.214341 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.214316    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 17:52:28.214341 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.214329    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:52:28.219877 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.219837    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm"]
Sep 10 17:52:28.339568 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.339522    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9532de-fe26-4f19-897c-fab50704ade8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm\" (UID: \"2c9532de-fe26-4f19-897c-fab50704ade8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm"
Sep 10 17:52:28.441025 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.440977    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9532de-fe26-4f19-897c-fab50704ade8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm\" (UID: \"2c9532de-fe26-4f19-897c-fab50704ade8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm"
Sep 10 17:52:28.441730 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.441703    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9532de-fe26-4f19-897c-fab50704ade8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm\" (UID: \"2c9532de-fe26-4f19-897c-fab50704ade8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm"
Sep 10 17:52:28.523673 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.523586    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm"
Sep 10 17:52:28.804353 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.804327    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm"]
Sep 10 17:52:28.805467 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:52:28.805444    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9532de_fe26_4f19_897c_fab50704ade8.slice/crio-260279fbc7258f4c6c0623f54f21082983a2b845d7c17844b72021e0404d4b3e WatchSource:0}: Error finding container 260279fbc7258f4c6c0623f54f21082983a2b845d7c17844b72021e0404d4b3e: Status 404 returned error can't find the container with id 260279fbc7258f4c6c0623f54f21082983a2b845d7c17844b72021e0404d4b3e
Sep 10 17:52:28.920859 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:28.920822    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm" event={"ID":"2c9532de-fe26-4f19-897c-fab50704ade8","Type":"ContainerStarted","Data":"260279fbc7258f4c6c0623f54f21082983a2b845d7c17844b72021e0404d4b3e"}
Sep 10 17:52:29.927579 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:29.927472    2552 generic.go:356] "Generic (PLEG): container finished" podID="2c9532de-fe26-4f19-897c-fab50704ade8" containerID="b57165a680efeb14278539157a180544c42b741341bf8b12561d917d6a2806a6" exitCode=0
Sep 10 17:52:29.927579 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:29.927561    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm" event={"ID":"2c9532de-fe26-4f19-897c-fab50704ade8","Type":"ContainerDied","Data":"b57165a680efeb14278539157a180544c42b741341bf8b12561d917d6a2806a6"}
Sep 10 17:52:31.069460 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:31.069430    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm"
Sep 10 17:52:31.164128 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:31.164090    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9532de-fe26-4f19-897c-fab50704ade8-maas-api-service-ca\") pod \"2c9532de-fe26-4f19-897c-fab50704ade8\" (UID: \"2c9532de-fe26-4f19-897c-fab50704ade8\") "
Sep 10 17:52:31.164531 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:31.164477    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9532de-fe26-4f19-897c-fab50704ade8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c9532de-fe26-4f19-897c-fab50704ade8" (UID: "2c9532de-fe26-4f19-897c-fab50704ade8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:52:31.265838 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:31.265735    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9532de-fe26-4f19-897c-fab50704ade8-maas-api-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:52:31.939962 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:31.939935    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm"
Sep 10 17:52:31.940143 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:31.939960    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm" event={"ID":"2c9532de-fe26-4f19-897c-fab50704ade8","Type":"ContainerDied","Data":"260279fbc7258f4c6c0623f54f21082983a2b845d7c17844b72021e0404d4b3e"}
Sep 10 17:52:31.940143 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:31.940004    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="260279fbc7258f4c6c0623f54f21082983a2b845d7c17844b72021e0404d4b3e"
Sep 10 17:52:39.282936 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:39.282889    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm"]
Sep 10 17:52:39.285754 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:39.285719    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7621a1-vckzm"]
Sep 10 17:52:40.803952 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:40.803919    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c9532de-fe26-4f19-897c-fab50704ade8" path="/var/lib/kubelet/pods/2c9532de-fe26-4f19-897c-fab50704ade8/volumes"
Sep 10 17:52:42.404553 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:42.404512    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:52:46.264428 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.264391    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"]
Sep 10 17:52:46.264909 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.264679    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn" podUID="746856a7-c5c2-4381-82ad-43e7eacb4424" containerName="istio-proxy" containerID="cri-o://d255eee6dd88ff32116b546ae82b3a4db8e2b1ab565b36791dc1205f4b5f75b0" gracePeriod=30
Sep 10 17:52:46.634017 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.633984    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8"]
Sep 10 17:52:46.634401 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.634389    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f01190d-3054-4beb-a025-b0a555ac105d" containerName="istio-proxy"
Sep 10 17:52:46.634524 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.634403    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f01190d-3054-4beb-a025-b0a555ac105d" containerName="istio-proxy"
Sep 10 17:52:46.634524 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.634421    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9532de-fe26-4f19-897c-fab50704ade8" containerName="revoke-keys"
Sep 10 17:52:46.634524 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.634427    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9532de-fe26-4f19-897c-fab50704ade8" containerName="revoke-keys"
Sep 10 17:52:46.634524 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.634504    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c9532de-fe26-4f19-897c-fab50704ade8" containerName="revoke-keys"
Sep 10 17:52:46.637744 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.637726    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8"
Sep 10 17:52:46.640559 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.640537    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:52:46.640666 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.640540    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 17:52:46.645334 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.645305    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8"]
Sep 10 17:52:46.718329 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.718286    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c168350b-a964-43d5-aee4-56d3ad063863-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8\" (UID: \"c168350b-a964-43d5-aee4-56d3ad063863\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8"
Sep 10 17:52:46.819887 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.819845    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c168350b-a964-43d5-aee4-56d3ad063863-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8\" (UID: \"c168350b-a964-43d5-aee4-56d3ad063863\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8"
Sep 10 17:52:46.820586 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.820565    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c168350b-a964-43d5-aee4-56d3ad063863-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8\" (UID: \"c168350b-a964-43d5-aee4-56d3ad063863\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8"
Sep 10 17:52:46.949295 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:46.949200    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8"
Sep 10 17:52:47.252188 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:47.252158    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8"]
Sep 10 17:52:47.254422 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:52:47.254392    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc168350b_a964_43d5_aee4_56d3ad063863.slice/crio-2d7dedd01f613aaed89b4750a80c6483a39230e06a72cc98dc0bc03b0415d844 WatchSource:0}: Error finding container 2d7dedd01f613aaed89b4750a80c6483a39230e06a72cc98dc0bc03b0415d844: Status 404 returned error can't find the container with id 2d7dedd01f613aaed89b4750a80c6483a39230e06a72cc98dc0bc03b0415d844
Sep 10 17:52:48.034068 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:48.034034    2552 generic.go:356] "Generic (PLEG): container finished" podID="c168350b-a964-43d5-aee4-56d3ad063863" containerID="350c1b23fe379d0648ac83526f60de4160e1927d5b7700cfb58c528b3bcda56f" exitCode=0
Sep 10 17:52:48.034457 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:48.034095    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8" event={"ID":"c168350b-a964-43d5-aee4-56d3ad063863","Type":"ContainerDied","Data":"350c1b23fe379d0648ac83526f60de4160e1927d5b7700cfb58c528b3bcda56f"}
Sep 10 17:52:48.034457 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:48.034134    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8" event={"ID":"c168350b-a964-43d5-aee4-56d3ad063863","Type":"ContainerStarted","Data":"2d7dedd01f613aaed89b4750a80c6483a39230e06a72cc98dc0bc03b0415d844"}
Sep 10 17:52:49.174364 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:49.174338    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8"
Sep 10 17:52:49.242340 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:49.242308    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c168350b-a964-43d5-aee4-56d3ad063863-maas-api-service-ca\") pod \"c168350b-a964-43d5-aee4-56d3ad063863\" (UID: \"c168350b-a964-43d5-aee4-56d3ad063863\") "
Sep 10 17:52:49.242692 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:49.242667    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c168350b-a964-43d5-aee4-56d3ad063863-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c168350b-a964-43d5-aee4-56d3ad063863" (UID: "c168350b-a964-43d5-aee4-56d3ad063863"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:52:49.344052 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:49.344020    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c168350b-a964-43d5-aee4-56d3ad063863-maas-api-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:52:50.046013 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:50.045978    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8"
Sep 10 17:52:50.046219 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:50.046024    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8" event={"ID":"c168350b-a964-43d5-aee4-56d3ad063863","Type":"ContainerDied","Data":"2d7dedd01f613aaed89b4750a80c6483a39230e06a72cc98dc0bc03b0415d844"}
Sep 10 17:52:50.046219 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:50.046102    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d7dedd01f613aaed89b4750a80c6483a39230e06a72cc98dc0bc03b0415d844"
Sep 10 17:52:51.523684 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.523658    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:52:51.663457 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.663423    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-podinfo\") pod \"746856a7-c5c2-4381-82ad-43e7eacb4424\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") "
Sep 10 17:52:51.663671 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.663477    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-token\") pod \"746856a7-c5c2-4381-82ad-43e7eacb4424\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") "
Sep 10 17:52:51.663671 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.663532    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746856a7-c5c2-4381-82ad-43e7eacb4424-istiod-ca-cert\") pod \"746856a7-c5c2-4381-82ad-43e7eacb4424\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") "
Sep 10 17:52:51.663671 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.663562    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-workload-socket\") pod \"746856a7-c5c2-4381-82ad-43e7eacb4424\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") "
Sep 10 17:52:51.663671 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.663584    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-data\") pod \"746856a7-c5c2-4381-82ad-43e7eacb4424\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") "
Sep 10 17:52:51.663671 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.663630    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-envoy\") pod \"746856a7-c5c2-4381-82ad-43e7eacb4424\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") "
Sep 10 17:52:51.663671 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.663663    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kpdq\" (UniqueName: \"kubernetes.io/projected/746856a7-c5c2-4381-82ad-43e7eacb4424-kube-api-access-6kpdq\") pod \"746856a7-c5c2-4381-82ad-43e7eacb4424\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") "
Sep 10 17:52:51.663998 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.663719    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-credential-socket\") pod \"746856a7-c5c2-4381-82ad-43e7eacb4424\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") "
Sep 10 17:52:51.663998 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.663781    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-workload-certs\") pod \"746856a7-c5c2-4381-82ad-43e7eacb4424\" (UID: \"746856a7-c5c2-4381-82ad-43e7eacb4424\") "
Sep 10 17:52:51.663998 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.663910    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-data" (OuterVolumeSpecName: "istio-data") pod "746856a7-c5c2-4381-82ad-43e7eacb4424" (UID: "746856a7-c5c2-4381-82ad-43e7eacb4424"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:52:51.664158 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.663984    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746856a7-c5c2-4381-82ad-43e7eacb4424-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "746856a7-c5c2-4381-82ad-43e7eacb4424" (UID: "746856a7-c5c2-4381-82ad-43e7eacb4424"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:52:51.664158 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.664108    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "746856a7-c5c2-4381-82ad-43e7eacb4424" (UID: "746856a7-c5c2-4381-82ad-43e7eacb4424"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:52:51.664267 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.664158    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "746856a7-c5c2-4381-82ad-43e7eacb4424" (UID: "746856a7-c5c2-4381-82ad-43e7eacb4424"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:52:51.664267 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.664218    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "746856a7-c5c2-4381-82ad-43e7eacb4424" (UID: "746856a7-c5c2-4381-82ad-43e7eacb4424"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:52:51.664403 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.664282    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-workload-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:52:51.664403 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.664309    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746856a7-c5c2-4381-82ad-43e7eacb4424-istiod-ca-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:52:51.664403 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.664322    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-workload-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:52:51.664403 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.664333    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-data\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:52:51.665768 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.665738    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-token" (OuterVolumeSpecName: "istio-token") pod "746856a7-c5c2-4381-82ad-43e7eacb4424" (UID: "746856a7-c5c2-4381-82ad-43e7eacb4424"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:52:51.666039 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.666012    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746856a7-c5c2-4381-82ad-43e7eacb4424-kube-api-access-6kpdq" (OuterVolumeSpecName: "kube-api-access-6kpdq") pod "746856a7-c5c2-4381-82ad-43e7eacb4424" (UID: "746856a7-c5c2-4381-82ad-43e7eacb4424"). InnerVolumeSpecName "kube-api-access-6kpdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:52:51.666192 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.666164    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "746856a7-c5c2-4381-82ad-43e7eacb4424" (UID: "746856a7-c5c2-4381-82ad-43e7eacb4424"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:52:51.666192 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.666177    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "746856a7-c5c2-4381-82ad-43e7eacb4424" (UID: "746856a7-c5c2-4381-82ad-43e7eacb4424"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:52:51.765183 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.765144    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-envoy\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:52:51.765183 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.765174    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kpdq\" (UniqueName: \"kubernetes.io/projected/746856a7-c5c2-4381-82ad-43e7eacb4424-kube-api-access-6kpdq\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:52:51.765183 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.765184    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746856a7-c5c2-4381-82ad-43e7eacb4424-credential-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:52:51.765183 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.765194    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-podinfo\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:52:51.765183 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:51.765202    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746856a7-c5c2-4381-82ad-43e7eacb4424-istio-token\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:52:52.060360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:52.060265    2552 generic.go:356] "Generic (PLEG): container finished" podID="746856a7-c5c2-4381-82ad-43e7eacb4424" containerID="d255eee6dd88ff32116b546ae82b3a4db8e2b1ab565b36791dc1205f4b5f75b0" exitCode=0
Sep 10 17:52:52.060360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:52.060319    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"
Sep 10 17:52:52.060360 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:52.060324    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn" event={"ID":"746856a7-c5c2-4381-82ad-43e7eacb4424","Type":"ContainerDied","Data":"d255eee6dd88ff32116b546ae82b3a4db8e2b1ab565b36791dc1205f4b5f75b0"}
Sep 10 17:52:52.060643 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:52.060382    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn" event={"ID":"746856a7-c5c2-4381-82ad-43e7eacb4424","Type":"ContainerDied","Data":"c8c84e4c41b949d0aa7f5aab190b12602643981b51e48db7cf9e997d04e1e5ca"}
Sep 10 17:52:52.060643 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:52.060411    2552 scope.go:117] "RemoveContainer" containerID="d255eee6dd88ff32116b546ae82b3a4db8e2b1ab565b36791dc1205f4b5f75b0"
Sep 10 17:52:52.070552 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:52.070530    2552 scope.go:117] "RemoveContainer" containerID="c2d178ecff82ee4b433cd6b3dc24159d7b7c9a4b85b2da67d16ddc240bc78799"
Sep 10 17:52:52.079132 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:52.079111    2552 scope.go:117] "RemoveContainer" containerID="d255eee6dd88ff32116b546ae82b3a4db8e2b1ab565b36791dc1205f4b5f75b0"
Sep 10 17:52:52.079413 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:52:52.079392    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d255eee6dd88ff32116b546ae82b3a4db8e2b1ab565b36791dc1205f4b5f75b0\": container with ID starting with d255eee6dd88ff32116b546ae82b3a4db8e2b1ab565b36791dc1205f4b5f75b0 not found: ID does not exist" containerID="d255eee6dd88ff32116b546ae82b3a4db8e2b1ab565b36791dc1205f4b5f75b0"
Sep 10 17:52:52.079463 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:52.079424    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d255eee6dd88ff32116b546ae82b3a4db8e2b1ab565b36791dc1205f4b5f75b0"} err="failed to get container status \"d255eee6dd88ff32116b546ae82b3a4db8e2b1ab565b36791dc1205f4b5f75b0\": rpc error: code = NotFound desc = could not find container \"d255eee6dd88ff32116b546ae82b3a4db8e2b1ab565b36791dc1205f4b5f75b0\": container with ID starting with d255eee6dd88ff32116b546ae82b3a4db8e2b1ab565b36791dc1205f4b5f75b0 not found: ID does not exist"
Sep 10 17:52:52.079463 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:52.079443    2552 scope.go:117] "RemoveContainer" containerID="c2d178ecff82ee4b433cd6b3dc24159d7b7c9a4b85b2da67d16ddc240bc78799"
Sep 10 17:52:52.079691 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:52:52.079676    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d178ecff82ee4b433cd6b3dc24159d7b7c9a4b85b2da67d16ddc240bc78799\": container with ID starting with c2d178ecff82ee4b433cd6b3dc24159d7b7c9a4b85b2da67d16ddc240bc78799 not found: ID does not exist" containerID="c2d178ecff82ee4b433cd6b3dc24159d7b7c9a4b85b2da67d16ddc240bc78799"
Sep 10 17:52:52.079729 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:52.079694    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d178ecff82ee4b433cd6b3dc24159d7b7c9a4b85b2da67d16ddc240bc78799"} err="failed to get container status \"c2d178ecff82ee4b433cd6b3dc24159d7b7c9a4b85b2da67d16ddc240bc78799\": rpc error: code = NotFound desc = could not find container \"c2d178ecff82ee4b433cd6b3dc24159d7b7c9a4b85b2da67d16ddc240bc78799\": container with ID starting with c2d178ecff82ee4b433cd6b3dc24159d7b7c9a4b85b2da67d16ddc240bc78799 not found: ID does not exist"
Sep 10 17:52:52.085521 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:52.085478    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"]
Sep 10 17:52:52.089919 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:52.089894    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7621a1-openshift-default-8574db548-nvdkn"]
Sep 10 17:52:52.803916 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:52.803886    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="746856a7-c5c2-4381-82ad-43e7eacb4424" path="/var/lib/kubelet/pods/746856a7-c5c2-4381-82ad-43e7eacb4424/volumes"
Sep 10 17:52:57.168016 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:57.167976    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-058069-29817705-zwbjf"]
Sep 10 17:52:57.172863 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:57.172836    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-058069-29817705-zwbjf"]
Sep 10 17:52:57.380169 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:57.380134    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8"]
Sep 10 17:52:57.383463 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:57.383434    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-058069-rg6w8"]
Sep 10 17:52:58.803752 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:58.803717    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13e8e355-cb5d-4680-9c09-acd387881d02" path="/var/lib/kubelet/pods/13e8e355-cb5d-4680-9c09-acd387881d02/volumes"
Sep 10 17:52:58.804130 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:52:58.804030    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c168350b-a964-43d5-aee4-56d3ad063863" path="/var/lib/kubelet/pods/c168350b-a964-43d5-aee4-56d3ad063863/volumes"
Sep 10 17:53:00.251541 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:00.251502    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:53:05.004601 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.004568    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk"]
Sep 10 17:53:05.005007 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.004959    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746856a7-c5c2-4381-82ad-43e7eacb4424" containerName="istio-proxy"
Sep 10 17:53:05.005007 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.004971    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="746856a7-c5c2-4381-82ad-43e7eacb4424" containerName="istio-proxy"
Sep 10 17:53:05.005007 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.004989    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c168350b-a964-43d5-aee4-56d3ad063863" containerName="revoke-keys"
Sep 10 17:53:05.005007 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.005003    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c168350b-a964-43d5-aee4-56d3ad063863" containerName="revoke-keys"
Sep 10 17:53:05.005151 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.005078    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c168350b-a964-43d5-aee4-56d3ad063863" containerName="revoke-keys"
Sep 10 17:53:05.005151 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.005090    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="746856a7-c5c2-4381-82ad-43e7eacb4424" containerName="istio-proxy"
Sep 10 17:53:05.005151 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.005096    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="746856a7-c5c2-4381-82ad-43e7eacb4424" containerName="istio-proxy"
Sep 10 17:53:05.009963 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.009942    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk"
Sep 10 17:53:05.012880 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.012856    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:53:05.013057 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.012901    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 17:53:05.016606 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.016582    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk"]
Sep 10 17:53:05.091758 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.091719    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk\" (UID: \"1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk"
Sep 10 17:53:05.192844 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.192810    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk\" (UID: \"1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk"
Sep 10 17:53:05.193450 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.193428    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk\" (UID: \"1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk"
Sep 10 17:53:05.323341 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.323252    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk"
Sep 10 17:53:05.596413 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:05.596385    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk"]
Sep 10 17:53:05.598246 ip-10-0-129-90 kubenswrapper[2552]: W0910 17:53:05.598221    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d68f390_f5c8_4f5b_82ce_c9c5ebab8fa7.slice/crio-2b203da36b7dcc09546d7ee90082b7eff77b6232f9718ec73956755a64b90dae WatchSource:0}: Error finding container 2b203da36b7dcc09546d7ee90082b7eff77b6232f9718ec73956755a64b90dae: Status 404 returned error can't find the container with id 2b203da36b7dcc09546d7ee90082b7eff77b6232f9718ec73956755a64b90dae
Sep 10 17:53:06.135275 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:06.135174    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk" event={"ID":"1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7","Type":"ContainerStarted","Data":"628222fe5e06d5c5a155a25746072e3d5a8ac04373f4d32781c8646b438c0dd6"}
Sep 10 17:53:06.135275 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:06.135238    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk" event={"ID":"1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7","Type":"ContainerStarted","Data":"2b203da36b7dcc09546d7ee90082b7eff77b6232f9718ec73956755a64b90dae"}
Sep 10 17:53:06.161120 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:06.161065    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk" podStartSLOduration=2.161048314 podStartE2EDuration="2.161048314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:53:06.15788679 +0000 UTC m=+1757.916413859" watchObservedRunningTime="2026-09-10 17:53:06.161048314 +0000 UTC m=+1757.919575384"
Sep 10 17:53:07.142148 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:07.142112    2552 generic.go:356] "Generic (PLEG): container finished" podID="1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7" containerID="628222fe5e06d5c5a155a25746072e3d5a8ac04373f4d32781c8646b438c0dd6" exitCode=0
Sep 10 17:53:07.142562 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:07.142208    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk" event={"ID":"1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7","Type":"ContainerDied","Data":"628222fe5e06d5c5a155a25746072e3d5a8ac04373f4d32781c8646b438c0dd6"}
Sep 10 17:53:08.283839 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:08.283812    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk"
Sep 10 17:53:08.316125 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:08.316095    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7-maas-api-service-ca\") pod \"1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7\" (UID: \"1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7\") "
Sep 10 17:53:08.316478 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:08.316454    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7" (UID: "1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:53:08.417226 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:08.417134    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7-maas-api-service-ca\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:53:09.155582 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:09.155552    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk"
Sep 10 17:53:09.155784 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:09.155547    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk" event={"ID":"1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7","Type":"ContainerDied","Data":"2b203da36b7dcc09546d7ee90082b7eff77b6232f9718ec73956755a64b90dae"}
Sep 10 17:53:09.155784 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:09.155697    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b203da36b7dcc09546d7ee90082b7eff77b6232f9718ec73956755a64b90dae"
Sep 10 17:53:15.596641 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:15.596601    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f14bf-29817705-s6nk8"]
Sep 10 17:53:15.599051 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:15.599023    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f14bf-29817705-s6nk8"]
Sep 10 17:53:15.803502 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:15.803463    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk"]
Sep 10 17:53:15.807255 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:15.807229    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f14bf-57phk"]
Sep 10 17:53:16.803596 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:16.803565    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7" path="/var/lib/kubelet/pods/1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7/volumes"
Sep 10 17:53:16.803964 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:16.803849    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e79bd00d-9e4d-4e35-86dc-e5e6670e7cc1" path="/var/lib/kubelet/pods/e79bd00d-9e4d-4e35-86dc-e5e6670e7cc1/volumes"
Sep 10 17:53:18.314700 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:18.314665    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:53:22.579301 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:22.579265    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"]
Sep 10 17:53:22.579736 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:22.579582    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs" podUID="1b52144b-be1b-47fd-a056-46f645501009" containerName="istio-proxy" containerID="cri-o://ebe5992f1eabbe6ea07988ebf80c98493418f9bfd025d1aee1dce8675324544b" gracePeriod=30
Sep 10 17:53:27.201882 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.201839    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:53:27.837432 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.837405    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:53:27.896266 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.896186    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-workload-certs\") pod \"1b52144b-be1b-47fd-a056-46f645501009\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") "
Sep 10 17:53:27.896266 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.896250    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv2j8\" (UniqueName: \"kubernetes.io/projected/1b52144b-be1b-47fd-a056-46f645501009-kube-api-access-sv2j8\") pod \"1b52144b-be1b-47fd-a056-46f645501009\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") "
Sep 10 17:53:27.896505 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.896304    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b52144b-be1b-47fd-a056-46f645501009-istio-podinfo\") pod \"1b52144b-be1b-47fd-a056-46f645501009\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") "
Sep 10 17:53:27.896505 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.896340    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-credential-socket\") pod \"1b52144b-be1b-47fd-a056-46f645501009\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") "
Sep 10 17:53:27.896505 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.896390    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-istio-envoy\") pod \"1b52144b-be1b-47fd-a056-46f645501009\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") "
Sep 10 17:53:27.896505 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.896413    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b52144b-be1b-47fd-a056-46f645501009-istiod-ca-cert\") pod \"1b52144b-be1b-47fd-a056-46f645501009\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") "
Sep 10 17:53:27.896505 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.896437    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1b52144b-be1b-47fd-a056-46f645501009" (UID: "1b52144b-be1b-47fd-a056-46f645501009"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:53:27.896505 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.896448    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-workload-socket\") pod \"1b52144b-be1b-47fd-a056-46f645501009\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") "
Sep 10 17:53:27.896837 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.896533    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-istio-data\") pod \"1b52144b-be1b-47fd-a056-46f645501009\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") "
Sep 10 17:53:27.896837 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.896593    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b52144b-be1b-47fd-a056-46f645501009-istio-token\") pod \"1b52144b-be1b-47fd-a056-46f645501009\" (UID: \"1b52144b-be1b-47fd-a056-46f645501009\") "
Sep 10 17:53:27.896837 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.896723    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1b52144b-be1b-47fd-a056-46f645501009" (UID: "1b52144b-be1b-47fd-a056-46f645501009"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:53:27.896837 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.896735    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1b52144b-be1b-47fd-a056-46f645501009" (UID: "1b52144b-be1b-47fd-a056-46f645501009"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:53:27.897062 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.896900    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-istio-data" (OuterVolumeSpecName: "istio-data") pod "1b52144b-be1b-47fd-a056-46f645501009" (UID: "1b52144b-be1b-47fd-a056-46f645501009"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:53:27.897062 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.897050    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-credential-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:53:27.897176 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.897070    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-workload-socket\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:53:27.897176 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.897083    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-istio-data\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:53:27.897176 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.897096    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-workload-certs\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:53:27.897176 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.897139    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b52144b-be1b-47fd-a056-46f645501009-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1b52144b-be1b-47fd-a056-46f645501009" (UID: "1b52144b-be1b-47fd-a056-46f645501009"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:53:27.898573 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.898544    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b52144b-be1b-47fd-a056-46f645501009-kube-api-access-sv2j8" (OuterVolumeSpecName: "kube-api-access-sv2j8") pod "1b52144b-be1b-47fd-a056-46f645501009" (UID: "1b52144b-be1b-47fd-a056-46f645501009"). InnerVolumeSpecName "kube-api-access-sv2j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:53:27.898782 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.898759    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b52144b-be1b-47fd-a056-46f645501009-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1b52144b-be1b-47fd-a056-46f645501009" (UID: "1b52144b-be1b-47fd-a056-46f645501009"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:53:27.899093 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.899062    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b52144b-be1b-47fd-a056-46f645501009-istio-token" (OuterVolumeSpecName: "istio-token") pod "1b52144b-be1b-47fd-a056-46f645501009" (UID: "1b52144b-be1b-47fd-a056-46f645501009"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:53:27.899093 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.899069    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1b52144b-be1b-47fd-a056-46f645501009" (UID: "1b52144b-be1b-47fd-a056-46f645501009"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:53:27.998411 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.998375    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b52144b-be1b-47fd-a056-46f645501009-istio-podinfo\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:53:27.998411 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.998404    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b52144b-be1b-47fd-a056-46f645501009-istio-envoy\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:53:27.998411 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.998413    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b52144b-be1b-47fd-a056-46f645501009-istiod-ca-cert\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:53:27.998411 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.998420    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b52144b-be1b-47fd-a056-46f645501009-istio-token\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:53:27.998707 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:27.998428    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv2j8\" (UniqueName: \"kubernetes.io/projected/1b52144b-be1b-47fd-a056-46f645501009-kube-api-access-sv2j8\") on node \"ip-10-0-129-90.ec2.internal\" DevicePath \"\""
Sep 10 17:53:28.257538 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:28.257426    2552 generic.go:356] "Generic (PLEG): container finished" podID="1b52144b-be1b-47fd-a056-46f645501009" containerID="ebe5992f1eabbe6ea07988ebf80c98493418f9bfd025d1aee1dce8675324544b" exitCode=0
Sep 10 17:53:28.257538 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:28.257480    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"
Sep 10 17:53:28.257976 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:28.257481    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs" event={"ID":"1b52144b-be1b-47fd-a056-46f645501009","Type":"ContainerDied","Data":"ebe5992f1eabbe6ea07988ebf80c98493418f9bfd025d1aee1dce8675324544b"}
Sep 10 17:53:28.257976 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:28.257625    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs" event={"ID":"1b52144b-be1b-47fd-a056-46f645501009","Type":"ContainerDied","Data":"63803e6e5a107e637bd9b3c170b07b7a6a76c682d35d6c9e4a87d1f24c479744"}
Sep 10 17:53:28.257976 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:28.257655    2552 scope.go:117] "RemoveContainer" containerID="ebe5992f1eabbe6ea07988ebf80c98493418f9bfd025d1aee1dce8675324544b"
Sep 10 17:53:28.267125 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:28.267111    2552 scope.go:117] "RemoveContainer" containerID="6f35d9c9dbf3b73b8e14582535a1683c588d938bf60218637d73aef89bfa45f3"
Sep 10 17:53:28.275026 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:28.275005    2552 scope.go:117] "RemoveContainer" containerID="ebe5992f1eabbe6ea07988ebf80c98493418f9bfd025d1aee1dce8675324544b"
Sep 10 17:53:28.275277 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:53:28.275259    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe5992f1eabbe6ea07988ebf80c98493418f9bfd025d1aee1dce8675324544b\": container with ID starting with ebe5992f1eabbe6ea07988ebf80c98493418f9bfd025d1aee1dce8675324544b not found: ID does not exist" containerID="ebe5992f1eabbe6ea07988ebf80c98493418f9bfd025d1aee1dce8675324544b"
Sep 10 17:53:28.275345 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:28.275283    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe5992f1eabbe6ea07988ebf80c98493418f9bfd025d1aee1dce8675324544b"} err="failed to get container status \"ebe5992f1eabbe6ea07988ebf80c98493418f9bfd025d1aee1dce8675324544b\": rpc error: code = NotFound desc = could not find container \"ebe5992f1eabbe6ea07988ebf80c98493418f9bfd025d1aee1dce8675324544b\": container with ID starting with ebe5992f1eabbe6ea07988ebf80c98493418f9bfd025d1aee1dce8675324544b not found: ID does not exist"
Sep 10 17:53:28.275345 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:28.275296    2552 scope.go:117] "RemoveContainer" containerID="6f35d9c9dbf3b73b8e14582535a1683c588d938bf60218637d73aef89bfa45f3"
Sep 10 17:53:28.275551 ip-10-0-129-90 kubenswrapper[2552]: E0910 17:53:28.275532    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f35d9c9dbf3b73b8e14582535a1683c588d938bf60218637d73aef89bfa45f3\": container with ID starting with 6f35d9c9dbf3b73b8e14582535a1683c588d938bf60218637d73aef89bfa45f3 not found: ID does not exist" containerID="6f35d9c9dbf3b73b8e14582535a1683c588d938bf60218637d73aef89bfa45f3"
Sep 10 17:53:28.275598 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:28.275558    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f35d9c9dbf3b73b8e14582535a1683c588d938bf60218637d73aef89bfa45f3"} err="failed to get container status \"6f35d9c9dbf3b73b8e14582535a1683c588d938bf60218637d73aef89bfa45f3\": rpc error: code = NotFound desc = could not find container \"6f35d9c9dbf3b73b8e14582535a1683c588d938bf60218637d73aef89bfa45f3\": container with ID starting with 6f35d9c9dbf3b73b8e14582535a1683c588d938bf60218637d73aef89bfa45f3 not found: ID does not exist"
Sep 10 17:53:28.281171 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:28.281149    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"]
Sep 10 17:53:28.286529 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:28.286484    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4f14bf-openshift-default-6489b97b4d-k8jhs"]
Sep 10 17:53:28.803572 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:28.803539    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b52144b-be1b-47fd-a056-46f645501009" path="/var/lib/kubelet/pods/1b52144b-be1b-47fd-a056-46f645501009/volumes"
Sep 10 17:53:48.825346 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:48.825310    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:53:48.831942 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:48.831918    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:53:49.072704 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:49.072673    2552 scope.go:117] "RemoveContainer" containerID="558b307a76feb862bedc429d38d8d8307fb743e66b79d12ec78166a7a6705cea"
Sep 10 17:53:49.081362 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:49.081340    2552 scope.go:117] "RemoveContainer" containerID="869ebad85b248ff09d41c6af892863ce481f3b8ed76e5d02cec7a4e9fbd5f443"
Sep 10 17:53:49.088972 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:53:49.088952    2552 scope.go:117] "RemoveContainer" containerID="d853cf30fb9ddda19f843d56717f41bbfc46fa971118f1d858ac21f1095fd34a"
Sep 10 17:54:47.303932 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:54:47.303893    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:54:49.125066 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:54:49.125033    2552 scope.go:117] "RemoveContainer" containerID="5c47a38b6dcd7cdfd3c65dde579abd4b231044e7ea028c691ec938ff70f02821"
Sep 10 17:54:49.133306 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:54:49.133282    2552 scope.go:117] "RemoveContainer" containerID="7f3c43b0469a6ab6bb77368b553ca11a14a01ef5daf9d41b34d9bdbb20d8f140"
Sep 10 17:55:11.902528 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:55:11.902424    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:55:19.812417 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:55:19.812377    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:55:28.202265 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:55:28.202225    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:55:45.800808 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:55:45.800767    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:55:52.496926 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:55:52.496886    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:56:49.163032 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:56:49.162999    2552 scope.go:117] "RemoveContainer" containerID="bc13cbefab065212042cd226dafc8679cd91b32575ecfc25cecdcfc16634b02e"
Sep 10 17:57:24.607456 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:57:24.607415    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:57:32.802784 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:57:32.802755    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:57:43.597724 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:57:43.597683    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:57:49.182550 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:57:49.182517    2552 scope.go:117] "RemoveContainer" containerID="4736e5cc4c7ffe2a0768f3f661c85bf84fb47bfae04b7d045570c2a9cac9684d"
Sep 10 17:57:50.908765 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:57:50.908723    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:58:45.309807 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:58:45.309765    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:58:48.856841 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:58:48.856813    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:58:48.865096 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:58:48.865068    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 17:58:49.202669 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:58:49.202576    2552 scope.go:117] "RemoveContainer" containerID="350c1b23fe379d0648ac83526f60de4160e1927d5b7700cfb58c528b3bcda56f"
Sep 10 17:58:49.212739 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:58:49.212714    2552 scope.go:117] "RemoveContainer" containerID="b57165a680efeb14278539157a180544c42b741341bf8b12561d917d6a2806a6"
Sep 10 17:58:53.608107 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:58:53.608066    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:59:01.993922 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:01.993870    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:59:08.997680 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:08.997641    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:59:15.800216 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:15.800181    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:59:23.903329 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:23.903286    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:59:29.105541 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:29.105489    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565f"]
Sep 10 17:59:31.785462 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:31.785431    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bf9c7b78-bb5jp_39eb76b8-26c0-41c7-ad67-6918247344e9/authorino/0.log"
Sep 10 17:59:37.753862 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:37.753832    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b6bb7766-wmbpk_7355930c-a88e-4edf-8eec-a8811ae65036/copy-manifests/0.log"
Sep 10 17:59:37.760894 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:37.760861    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b6bb7766-wmbpk_7355930c-a88e-4edf-8eec-a8811ae65036/manager/0.log"
Sep 10 17:59:38.469476 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:38.469444    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817705-9c6s2_3d41edb2-413f-477e-bdd1-af69030b05d0/cleanup/0.log"
Sep 10 17:59:39.388129 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:39.388095    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bf9c7b78-bb5jp_39eb76b8-26c0-41c7-ad67-6918247344e9/authorino/0.log"
Sep 10 17:59:39.742371 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:39.742293    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-btmx5_3e67cb1b-0e95-45db-9d3b-979e933acfbf/kuadrant-console-plugin/0.log"
Sep 10 17:59:39.983777 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:39.983746    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8565f_44ec4c7e-fc85-4d18-a150-bc77a6b1a65c/limitador/0.log"
Sep 10 17:59:40.680669 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:40.680635    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bf54bbc88-gb5nr_6fd72f14-7331-4679-9cf6-c29e3c9222f6/kube-auth-proxy/0.log"
Sep 10 17:59:41.620478 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:41.620447    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-f7nnm_9c8821e3-f435-4e87-844b-2ee6a2fea0ec/main/0.log"
Sep 10 17:59:41.734687 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:41.734659    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-j7l6n_9e1eec56-17e5-4cb1-8628-ce81e0ea0b16/main/0.log"
Sep 10 17:59:42.291305 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:42.291274    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-4r9m5_dc851fb4-2a5c-40d9-8f5f-084aee0dae96/main/0.log"
Sep 10 17:59:49.234416 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:49.234382    2552 scope.go:117] "RemoveContainer" containerID="628222fe5e06d5c5a155a25746072e3d5a8ac04373f4d32781c8646b438c0dd6"
Sep 10 17:59:50.880315 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:50.880284    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bbbdj_ed270dc5-1f93-4739-b3b1-d5a1412d6819/global-pull-secret-syncer/0.log"
Sep 10 17:59:51.009343 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:51.009314    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ndkbc_17564146-f078-4b05-a08f-2ac7d4f1d60d/konnectivity-agent/0.log"
Sep 10 17:59:51.065359 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:51.065330    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-90.ec2.internal_9967cb0cd42cd2e177721f6d62bd6a5e/haproxy/0.log"
Sep 10 17:59:55.884771 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:55.884735    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bf9c7b78-bb5jp_39eb76b8-26c0-41c7-ad67-6918247344e9/authorino/0.log"
Sep 10 17:59:55.988846 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:55.988817    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-btmx5_3e67cb1b-0e95-45db-9d3b-979e933acfbf/kuadrant-console-plugin/0.log"
Sep 10 17:59:56.068467 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:56.068437    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8565f_44ec4c7e-fc85-4d18-a150-bc77a6b1a65c/limitador/0.log"
Sep 10 17:59:57.522649 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:57.522622    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_98993b00-3035-42d4-b263-7dc9aaeaa839/alertmanager/0.log"
Sep 10 17:59:57.562391 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:57.562367    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_98993b00-3035-42d4-b263-7dc9aaeaa839/config-reloader/0.log"
Sep 10 17:59:57.584877 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:57.584847    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_98993b00-3035-42d4-b263-7dc9aaeaa839/kube-rbac-proxy-web/0.log"
Sep 10 17:59:57.607287 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:57.607255    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_98993b00-3035-42d4-b263-7dc9aaeaa839/kube-rbac-proxy/0.log"
Sep 10 17:59:57.630645 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:57.630623    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_98993b00-3035-42d4-b263-7dc9aaeaa839/kube-rbac-proxy-metric/0.log"
Sep 10 17:59:57.656869 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:57.656851    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_98993b00-3035-42d4-b263-7dc9aaeaa839/prom-label-proxy/0.log"
Sep 10 17:59:57.678202 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:57.678182    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_98993b00-3035-42d4-b263-7dc9aaeaa839/init-config-reloader/0.log"
Sep 10 17:59:57.900600 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:57.900576    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-869f7667f5-qbtf8_7312f60e-373d-49fa-8213-c609144cd5df/metrics-server/0.log"
Sep 10 17:59:58.099598 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.099563    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mlz6w_85a41d92-3f00-4000-96a2-9d49c60196be/node-exporter/0.log"
Sep 10 17:59:58.124894 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.124813    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mlz6w_85a41d92-3f00-4000-96a2-9d49c60196be/kube-rbac-proxy/0.log"
Sep 10 17:59:58.145759 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.145728    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mlz6w_85a41d92-3f00-4000-96a2-9d49c60196be/init-textfile/0.log"
Sep 10 17:59:58.368381 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.368354    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_68bb510b-a2db-49b9-a21b-a9ab2007ab85/prometheus/0.log"
Sep 10 17:59:58.409432 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.409360    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_68bb510b-a2db-49b9-a21b-a9ab2007ab85/config-reloader/0.log"
Sep 10 17:59:58.432126 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.432100    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_68bb510b-a2db-49b9-a21b-a9ab2007ab85/thanos-sidecar/0.log"
Sep 10 17:59:58.452731 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.452710    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_68bb510b-a2db-49b9-a21b-a9ab2007ab85/kube-rbac-proxy-web/0.log"
Sep 10 17:59:58.480476 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.480453    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_68bb510b-a2db-49b9-a21b-a9ab2007ab85/kube-rbac-proxy/0.log"
Sep 10 17:59:58.503565 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.503543    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_68bb510b-a2db-49b9-a21b-a9ab2007ab85/kube-rbac-proxy-thanos/0.log"
Sep 10 17:59:58.530286 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.530262    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_68bb510b-a2db-49b9-a21b-a9ab2007ab85/init-config-reloader/0.log"
Sep 10 17:59:58.602614 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.602587    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-n9bcl_fb049c3e-b1ef-43b0-a466-18cd3d49575c/prometheus-operator-admission-webhook/0.log"
Sep 10 17:59:58.750183 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.750109    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9766557-xzzlt_fd3cb3c3-0799-45d5-8a45-776823e46cec/thanos-query/0.log"
Sep 10 17:59:58.771569 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.771544    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9766557-xzzlt_fd3cb3c3-0799-45d5-8a45-776823e46cec/kube-rbac-proxy-web/0.log"
Sep 10 17:59:58.791664 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.791641    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9766557-xzzlt_fd3cb3c3-0799-45d5-8a45-776823e46cec/kube-rbac-proxy/0.log"
Sep 10 17:59:58.812713 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.812687    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9766557-xzzlt_fd3cb3c3-0799-45d5-8a45-776823e46cec/prom-label-proxy/0.log"
Sep 10 17:59:58.833020 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.832999    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9766557-xzzlt_fd3cb3c3-0799-45d5-8a45-776823e46cec/kube-rbac-proxy-rules/0.log"
Sep 10 17:59:58.852582 ip-10-0-129-90 kubenswrapper[2552]: I0910 17:59:58.852556    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f9766557-xzzlt_fd3cb3c3-0799-45d5-8a45-776823e46cec/kube-rbac-proxy-metrics/0.log"
Sep 10 18:00:00.140314 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.140278    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817720-lrc44"]
Sep 10 18:00:00.140698 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.140648    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746856a7-c5c2-4381-82ad-43e7eacb4424" containerName="istio-proxy"
Sep 10 18:00:00.140698 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.140660    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="746856a7-c5c2-4381-82ad-43e7eacb4424" containerName="istio-proxy"
Sep 10 18:00:00.140698 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.140679    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7" containerName="revoke-keys"
Sep 10 18:00:00.140698 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.140684    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7" containerName="revoke-keys"
Sep 10 18:00:00.140830 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.140702    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b52144b-be1b-47fd-a056-46f645501009" containerName="istio-proxy"
Sep 10 18:00:00.140830 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.140708    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b52144b-be1b-47fd-a056-46f645501009" containerName="istio-proxy"
Sep 10 18:00:00.140830 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.140715    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b52144b-be1b-47fd-a056-46f645501009" containerName="istio-proxy"
Sep 10 18:00:00.140830 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.140721    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b52144b-be1b-47fd-a056-46f645501009" containerName="istio-proxy"
Sep 10 18:00:00.140830 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.140777    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b52144b-be1b-47fd-a056-46f645501009" containerName="istio-proxy"
Sep 10 18:00:00.140830 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.140787    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b52144b-be1b-47fd-a056-46f645501009" containerName="istio-proxy"
Sep 10 18:00:00.140830 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.140795    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d68f390-f5c8-4f5b-82ce-c9c5ebab8fa7" containerName="revoke-keys"
Sep 10 18:00:00.144058 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.144042    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817720-lrc44"
Sep 10 18:00:00.146768 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.146749    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77v6x\""
Sep 10 18:00:00.152373 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.152349    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817720-lrc44"]
Sep 10 18:00:00.155110 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.155087    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817720-lrc44"
Sep 10 18:00:00.219934 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.219102    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-clnst_29057183-7c39-4679-80cd-aa4d4c2deebe/networking-console-plugin/0.log"
Sep 10 18:00:00.424879 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.424852    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817720-lrc44"]
Sep 10 18:00:00.426980 ip-10-0-129-90 kubenswrapper[2552]: W0910 18:00:00.426946    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c870b42_c25c_4e28_8d36_1cdf6adb04bc.slice/crio-49b8772b4fd2bc69d8f71b6dafc9f0887e9cabb21701b26f7c8fc1254cb9a60a WatchSource:0}: Error finding container 49b8772b4fd2bc69d8f71b6dafc9f0887e9cabb21701b26f7c8fc1254cb9a60a: Status 404 returned error can't find the container with id 49b8772b4fd2bc69d8f71b6dafc9f0887e9cabb21701b26f7c8fc1254cb9a60a
Sep 10 18:00:00.802517 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.802467    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"]
Sep 10 18:00:00.805902 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.805885    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:00.809116 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.809089    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8jqk\"/\"openshift-service-ca.crt\""
Sep 10 18:00:00.809116 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.809098    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8jqk\"/\"kube-root-ca.crt\""
Sep 10 18:00:00.813441 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.813419    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j8jqk\"/\"default-dockercfg-hcms2\""
Sep 10 18:00:00.817101 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.817077    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"]
Sep 10 18:00:00.855112 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.855084    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/1.log"
Sep 10 18:00:00.860741 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.860714    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mwgvb_89a48e5b-848f-46fa-a181-4d4b331761be/console-operator/2.log"
Sep 10 18:00:00.921378 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.921337    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fmxr\" (UniqueName: \"kubernetes.io/projected/0275d3eb-673d-4730-8daf-df4fdbed0a34-kube-api-access-8fmxr\") pod \"perf-node-gather-daemonset-tz5vq\" (UID: \"0275d3eb-673d-4730-8daf-df4fdbed0a34\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:00.921574 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.921390    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0275d3eb-673d-4730-8daf-df4fdbed0a34-lib-modules\") pod \"perf-node-gather-daemonset-tz5vq\" (UID: \"0275d3eb-673d-4730-8daf-df4fdbed0a34\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:00.921574 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.921436    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0275d3eb-673d-4730-8daf-df4fdbed0a34-proc\") pod \"perf-node-gather-daemonset-tz5vq\" (UID: \"0275d3eb-673d-4730-8daf-df4fdbed0a34\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:00.921574 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.921482    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0275d3eb-673d-4730-8daf-df4fdbed0a34-sys\") pod \"perf-node-gather-daemonset-tz5vq\" (UID: \"0275d3eb-673d-4730-8daf-df4fdbed0a34\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:00.921574 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:00.921546    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0275d3eb-673d-4730-8daf-df4fdbed0a34-podres\") pod \"perf-node-gather-daemonset-tz5vq\" (UID: \"0275d3eb-673d-4730-8daf-df4fdbed0a34\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:01.023076 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.022964    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0275d3eb-673d-4730-8daf-df4fdbed0a34-lib-modules\") pod \"perf-node-gather-daemonset-tz5vq\" (UID: \"0275d3eb-673d-4730-8daf-df4fdbed0a34\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:01.023076 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.023020    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0275d3eb-673d-4730-8daf-df4fdbed0a34-proc\") pod \"perf-node-gather-daemonset-tz5vq\" (UID: \"0275d3eb-673d-4730-8daf-df4fdbed0a34\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:01.023076 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.023060    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0275d3eb-673d-4730-8daf-df4fdbed0a34-sys\") pod \"perf-node-gather-daemonset-tz5vq\" (UID: \"0275d3eb-673d-4730-8daf-df4fdbed0a34\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:01.023301 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.023096    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0275d3eb-673d-4730-8daf-df4fdbed0a34-podres\") pod \"perf-node-gather-daemonset-tz5vq\" (UID: \"0275d3eb-673d-4730-8daf-df4fdbed0a34\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:01.023301 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.023162    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fmxr\" (UniqueName: \"kubernetes.io/projected/0275d3eb-673d-4730-8daf-df4fdbed0a34-kube-api-access-8fmxr\") pod \"perf-node-gather-daemonset-tz5vq\" (UID: \"0275d3eb-673d-4730-8daf-df4fdbed0a34\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:01.023301 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.023190    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0275d3eb-673d-4730-8daf-df4fdbed0a34-sys\") pod \"perf-node-gather-daemonset-tz5vq\" (UID: \"0275d3eb-673d-4730-8daf-df4fdbed0a34\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:01.023301 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.023187    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0275d3eb-673d-4730-8daf-df4fdbed0a34-proc\") pod \"perf-node-gather-daemonset-tz5vq\" (UID: \"0275d3eb-673d-4730-8daf-df4fdbed0a34\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:01.023301 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.023187    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0275d3eb-673d-4730-8daf-df4fdbed0a34-lib-modules\") pod \"perf-node-gather-daemonset-tz5vq\" (UID: \"0275d3eb-673d-4730-8daf-df4fdbed0a34\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:01.023301 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.023254    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0275d3eb-673d-4730-8daf-df4fdbed0a34-podres\") pod \"perf-node-gather-daemonset-tz5vq\" (UID: \"0275d3eb-673d-4730-8daf-df4fdbed0a34\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:01.031968 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.031937    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fmxr\" (UniqueName: \"kubernetes.io/projected/0275d3eb-673d-4730-8daf-df4fdbed0a34-kube-api-access-8fmxr\") pod \"perf-node-gather-daemonset-tz5vq\" (UID: \"0275d3eb-673d-4730-8daf-df4fdbed0a34\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:01.116823 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.116785    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:01.218810 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.218773    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817720-lrc44" event={"ID":"1c870b42-c25c-4e28-8d36-1cdf6adb04bc","Type":"ContainerStarted","Data":"61ab95d7c97821ac1a6496fbfe9a26c0ebf0da829ea71c8c76b8e1e6c0c4ab0c"}
Sep 10 18:00:01.219189 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.218824    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817720-lrc44" event={"ID":"1c870b42-c25c-4e28-8d36-1cdf6adb04bc","Type":"ContainerStarted","Data":"49b8772b4fd2bc69d8f71b6dafc9f0887e9cabb21701b26f7c8fc1254cb9a60a"}
Sep 10 18:00:01.234576 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.234528    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817720-lrc44" podStartSLOduration=1.234514281 podStartE2EDuration="1.234514281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:00:01.233547591 +0000 UTC m=+2172.992074665" watchObservedRunningTime="2026-09-10 18:00:01.234514281 +0000 UTC m=+2172.993041341"
Sep 10 18:00:01.396819 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.396738    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"]
Sep 10 18:00:01.399538 ip-10-0-129-90 kubenswrapper[2552]: W0910 18:00:01.399510    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0275d3eb_673d_4730_8daf_df4fdbed0a34.slice/crio-f1270e6daac3c0fd71db14be44a68a6ae2e581fd7dc62a45e7e8cc39176a484d WatchSource:0}: Error finding container f1270e6daac3c0fd71db14be44a68a6ae2e581fd7dc62a45e7e8cc39176a484d: Status 404 returned error can't find the container with id f1270e6daac3c0fd71db14be44a68a6ae2e581fd7dc62a45e7e8cc39176a484d
Sep 10 18:00:01.459937 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:01.459914    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-ccl54_59a60e4b-21b8-4ede-b64c-8164dc12742c/download-server/0.log"
Sep 10 18:00:02.224822 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:02.224787    2552 generic.go:356] "Generic (PLEG): container finished" podID="1c870b42-c25c-4e28-8d36-1cdf6adb04bc" containerID="61ab95d7c97821ac1a6496fbfe9a26c0ebf0da829ea71c8c76b8e1e6c0c4ab0c" exitCode=0
Sep 10 18:00:02.225253 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:02.224868    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817720-lrc44" event={"ID":"1c870b42-c25c-4e28-8d36-1cdf6adb04bc","Type":"ContainerDied","Data":"61ab95d7c97821ac1a6496fbfe9a26c0ebf0da829ea71c8c76b8e1e6c0c4ab0c"}
Sep 10 18:00:02.226413 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:02.226387    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq" event={"ID":"0275d3eb-673d-4730-8daf-df4fdbed0a34","Type":"ContainerStarted","Data":"1c834f0af495f15a60e8bc5f3c268e178c8490006a80e623a2c943e7e11c4949"}
Sep 10 18:00:02.226575 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:02.226437    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq" event={"ID":"0275d3eb-673d-4730-8daf-df4fdbed0a34","Type":"ContainerStarted","Data":"f1270e6daac3c0fd71db14be44a68a6ae2e581fd7dc62a45e7e8cc39176a484d"}
Sep 10 18:00:02.226575 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:02.226465    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:02.257037 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:02.256966    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq" podStartSLOduration=2.256945362 podStartE2EDuration="2.256945362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:00:02.256779164 +0000 UTC m=+2174.015306235" watchObservedRunningTime="2026-09-10 18:00:02.256945362 +0000 UTC m=+2174.015472433"
Sep 10 18:00:03.337120 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:03.337080    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-77x6z_a3b1670d-12c0-4614-a86e-22c5498e1274/dns/0.log"
Sep 10 18:00:03.358965 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:03.358941    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-77x6z_a3b1670d-12c0-4614-a86e-22c5498e1274/kube-rbac-proxy/0.log"
Sep 10 18:00:03.362101 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:03.362077    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817720-lrc44"
Sep 10 18:00:03.560127 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:03.560037    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f9tg8_81246f30-ad68-472c-aa8b-72d14634f501/dns-node-resolver/0.log"
Sep 10 18:00:04.151193 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:04.151157    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wwbzr_7067d92a-5363-42fe-a8f0-7c765931c532/node-ca/0.log"
Sep 10 18:00:04.237033 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:04.237004    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817720-lrc44"
Sep 10 18:00:04.237207 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:04.237000    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817720-lrc44" event={"ID":"1c870b42-c25c-4e28-8d36-1cdf6adb04bc","Type":"ContainerDied","Data":"49b8772b4fd2bc69d8f71b6dafc9f0887e9cabb21701b26f7c8fc1254cb9a60a"}
Sep 10 18:00:04.237207 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:04.237137    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49b8772b4fd2bc69d8f71b6dafc9f0887e9cabb21701b26f7c8fc1254cb9a60a"
Sep 10 18:00:05.536049 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:05.536019    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bf54bbc88-gb5nr_6fd72f14-7331-4679-9cf6-c29e3c9222f6/kube-auth-proxy/0.log"
Sep 10 18:00:06.630542 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:06.630513    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-55x6q_fff24ddc-90d7-4565-8f85-27c6f028cca4/serve-healthcheck-canary/0.log"
Sep 10 18:00:07.187261 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:07.187226    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-glx7s_6d163bbc-7b58-4c6d-9ced-45db33946115/kube-rbac-proxy/0.log"
Sep 10 18:00:07.209698 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:07.209669    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-glx7s_6d163bbc-7b58-4c6d-9ced-45db33946115/exporter/0.log"
Sep 10 18:00:07.229257 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:07.229232    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-glx7s_6d163bbc-7b58-4c6d-9ced-45db33946115/extractor/0.log"
Sep 10 18:00:08.241162 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:08.241135    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-tz5vq"
Sep 10 18:00:10.163964 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:10.163936    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b6bb7766-wmbpk_7355930c-a88e-4edf-8eec-a8811ae65036/manager/0.log"
Sep 10 18:00:10.188722 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:10.188698    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b6bb7766-wmbpk_7355930c-a88e-4edf-8eec-a8811ae65036/copy-manifests/0.log"
Sep 10 18:00:11.621763 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:11.621737    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-86c5cb66bd-w4qw9_ca0845e0-a958-4783-aa8d-3a1455aa0337/manager/0.log"
Sep 10 18:00:11.732433 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:11.732394    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bf54bbc88-gb5nr_6fd72f14-7331-4679-9cf6-c29e3c9222f6/kube-auth-proxy/0.log"
Sep 10 18:00:13.825623 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:13.825590    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bf54bbc88-gb5nr_6fd72f14-7331-4679-9cf6-c29e3c9222f6/kube-auth-proxy/0.log"
Sep 10 18:00:18.140542 ip-10-0-129-90 kubenswrapper[2552]: I0910 18:00:18.140510    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8h6xk_e992abfb-f6c8-410a-869e-7e7394580e7d/kube-multus/0.log"