Sep 04 23:36:56.228902 ip-10-0-142-236 systemd[1]: Starting Kubernetes Kubelet...
Sep 04 23:36:56.612497 ip-10-0-142-236 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 04 23:36:56.612497 ip-10-0-142-236 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 04 23:36:56.612497 ip-10-0-142-236 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 04 23:36:56.612497 ip-10-0-142-236 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 04 23:36:56.612497 ip-10-0-142-236 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 04 23:36:56.614123 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.614046    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 04 23:36:56.619152 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619134    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 23:36:56.619152 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619152    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 23:36:56.619152 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619157    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619163    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619167    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619171    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619175    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619178    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619184    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619188    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619192    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619196    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619200    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619204    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619207    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619212    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619216    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619220    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619224    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619228    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619232    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619236    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 23:36:56.619336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619240    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619252    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619257    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619260    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619265    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619269    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619273    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619278    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619282    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619286    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619290    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619294    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619298    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619302    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619307    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619311    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619315    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619320    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619324    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619328    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 23:36:56.620179 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619332    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619336    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619344    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619350    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619354    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619359    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619363    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619367    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619371    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619375    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619380    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619384    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619388    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619392    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619396    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619400    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619404    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619409    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619413    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619418    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 23:36:56.621031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619422    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619426    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619430    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619434    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619438    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619442    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619452    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619458    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619463    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619468    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619472    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619476    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619481    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619485    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619489    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619495    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619499    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619503    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619507    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 23:36:56.621909 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619511    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619516    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619522    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619527    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619533    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619538    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619543    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619547    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619551    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619555    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619560    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619564    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619569    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619573    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619577    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619598    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619602    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.619606    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620774    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 23:36:56.622732 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620785    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620790    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620795    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620799    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620804    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620809    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620813    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620817    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620821    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620825    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620830    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620835    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620839    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620846    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620853    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620857    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620862    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620867    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620872    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 23:36:56.623295 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620876    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620880    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620885    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620889    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620893    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620897    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620902    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620906    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620910    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620915    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620919    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620923    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620928    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620932    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620936    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620940    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620944    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620948    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620952    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620957    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 23:36:56.623830 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620962    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620966    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620970    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620974    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620979    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620983    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620987    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620991    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.620995    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621000    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621004    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621008    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621012    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621016    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621020    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621025    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621028    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621032    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621038    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621041    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 23:36:56.624318 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621046    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621049    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621054    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621058    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621062    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621069    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621073    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621077    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621081    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621085    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621089    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621094    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621098    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621102    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621106    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621110    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621114    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621119    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621126    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621131    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 23:36:56.624922 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621135    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621139    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621143    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621149    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621155    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621160    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621164    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621168    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621173    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621178    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621183    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621187    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621191    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621195    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621199    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621204    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621208    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621212    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621216    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621812    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 23:36:56.625460 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621821    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621826    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621830    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621835    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621839    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621843    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621848    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621852    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621856    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621861    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621865    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621870    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621875    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621879    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621883    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621888    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621893    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621897    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621901    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621905    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 23:36:56.626024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621909    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621913    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621917    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621921    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621927    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621931    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621936    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621942    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621948    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621954    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621959    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621964    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621968    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621972    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621976    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621981    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621985    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621989    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621994    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.621998    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 23:36:56.626558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622002    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622006    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622011    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622015    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622019    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622023    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622028    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622032    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622036    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622040    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622044    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622048    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622055    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622059    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622064    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622068    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622072    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622076    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622080    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622086    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 23:36:56.627108 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622090    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622094    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622099    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622102    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622107    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622113    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622119    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622123    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622128    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622132    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622137    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622142    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622146    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622150    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622154    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622159    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622163    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622168    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622172    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 23:36:56.627616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622176    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622180    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622184    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622189    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622193    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622199    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622204    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622208    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622213    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622219    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622224    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622228    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622232    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622236    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622241    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622245    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622249    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622253    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.622257    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622360    2557 flags.go:64] FLAG: --address="0.0.0.0"
Sep 04 23:36:56.628089 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622416    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622425    2557 flags.go:64] FLAG: --anonymous-auth="true"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622432    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622438    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622443    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622450    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622456    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622461    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622466    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622474    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622479    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622484    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622489    2557 flags.go:64] FLAG: --cgroup-root=""
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622494    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622500    2557 flags.go:64] FLAG: --client-ca-file=""
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622505    2557 flags.go:64] FLAG: --cloud-provider="external"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622510    2557 flags.go:64] FLAG: --cluster-dns="[]"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622518    2557 flags.go:64] FLAG: --cluster-domain=""
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622523    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622528    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622533    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622538    2557 flags.go:64] FLAG: --container-log-max-files="5"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622544    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 04 23:36:56.628664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622549    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622554    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622559    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622564    2557 flags.go:64] FLAG: --contention-profiling="false"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622568    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622574    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622597    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622602    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622613    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622617    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622623    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622627    2557 flags.go:64] FLAG: --enable-load-reader="false"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622632    2557 flags.go:64] FLAG: --enable-server="true"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622637    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622643    2557 flags.go:64] FLAG: --event-burst="100"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622649    2557 flags.go:64] FLAG: --event-qps="50"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622653    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622658    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622663    2557 flags.go:64] FLAG: --eviction-hard=""
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622670    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622674    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622679    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622684    2557 flags.go:64] FLAG: --eviction-soft=""
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622691    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622695    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 04 23:36:56.629632 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622700    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622706    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622711    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622716    2557 flags.go:64] FLAG: --fail-swap-on="true"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622720    2557 flags.go:64] FLAG: --feature-gates=""
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622726    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622730    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622735    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622740    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622744    2557 flags.go:64] FLAG: --healthz-port="10248"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622749    2557 flags.go:64] FLAG: --help="false"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622754    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-142-236.ec2.internal"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622759    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622764    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622768    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622774    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622780    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622784    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622790    2557 flags.go:64] FLAG: --image-service-endpoint=""
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622794    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622799    2557 flags.go:64] FLAG: --kube-api-burst="100"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622803    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622809    2557 flags.go:64] FLAG: --kube-api-qps="50"
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622813    2557 flags.go:64] FLAG: --kube-reserved=""
Sep 04 23:36:56.630741 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622818    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622823    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622828    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622833    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622838    2557 flags.go:64] FLAG: --lock-file=""
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622843    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622848    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622853    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622861    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622867    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622872    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622877    2557 flags.go:64] FLAG: --logging-format="text"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622881    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622886    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622891    2557 flags.go:64] FLAG: --manifest-url=""
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622895    2557 flags.go:64] FLAG: --manifest-url-header=""
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622902    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622907    2557 flags.go:64] FLAG: --max-open-files="1000000"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622913    2557 flags.go:64] FLAG: --max-pods="110"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622918    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622922    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622927    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622931    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622936    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622941    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 04 23:36:56.631778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622970    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622993    2557 flags.go:64] FLAG: --node-status-max-images="50"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.622998    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623002    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623005    2557 flags.go:64] FLAG: --pod-cidr=""
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623009    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623016    2557 flags.go:64] FLAG: --pod-manifest-path=""
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623019    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623023    2557 flags.go:64] FLAG: --pods-per-core="0"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623026    2557 flags.go:64] FLAG: --port="10250"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623029    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623033    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0aefcbbe01d5491b0"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623036    2557 flags.go:64] FLAG: --qos-reserved=""
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623039    2557 flags.go:64] FLAG: --read-only-port="10255"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623043    2557 flags.go:64] FLAG: --register-node="true"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623046    2557 flags.go:64] FLAG: --register-with-taints=""
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623050    2557 flags.go:64] FLAG: --registry-burst="10"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623052    2557 flags.go:64] FLAG: --registry-qps="5"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623055    2557 flags.go:64] FLAG: --reserved-cpus=""
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623058    2557 flags.go:64] FLAG: --reserved-memory=""
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623062    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623065    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623068    2557 flags.go:64] FLAG: --rotate-certificates="false"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623071    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623074    2557 flags.go:64] FLAG: --runonce="false"
Sep 04 23:36:56.632917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623076    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623080    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623083    2557 flags.go:64] FLAG: --seccomp-default="false"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623086    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623089    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623092    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623095    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623098    2557 flags.go:64] FLAG: --storage-driver-password="root"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623102    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623106    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623109    2557 flags.go:64] FLAG: --storage-driver-user="root"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623111    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623114    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623117    2557 flags.go:64] FLAG: --system-cgroups=""
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623121    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623134    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623137    2557 flags.go:64] FLAG: --tls-cert-file=""
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623140    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623144    2557 flags.go:64] FLAG: --tls-min-version=""
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623147    2557 flags.go:64] FLAG: --tls-private-key-file=""
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623150    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623153    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623156    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623159    2557 flags.go:64] FLAG: --v="2"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623163    2557 flags.go:64] FLAG: --version="false"
Sep 04 23:36:56.633683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623167    2557 flags.go:64] FLAG: --vmodule=""
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623171    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623174    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623292    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623296    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623299    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623303    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623307    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623310    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623312    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623315    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623318    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623320    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623323    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623326    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623329    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623332    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623335    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623338    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623341    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 23:36:56.634779 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623343    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623346    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623349    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623351    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623354    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623357    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623359    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623362    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623365    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623367    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623370    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623372    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623375    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623377    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623380    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623383    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623386    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623388    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623391    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623395    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 23:36:56.635425 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623399    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623403    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623406    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623409    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623411    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623414    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623416    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623419    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623422    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623426    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623431    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623434    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623437    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623440    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623443    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623446    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623448    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623451    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623453    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 23:36:56.636088 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623456    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623458    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623461    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623463    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623466    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623468    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623471    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623473    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623476    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623478    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623481    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623484    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623486    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623489    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623491    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623494    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623496    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623499    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623502    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623504    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 23:36:56.636561 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623509    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623514    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623517    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623520    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623522    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623525    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623527    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623530    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623532    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623535    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623537    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623540    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623543    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623545    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623548    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623550    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623552    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623555    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623558    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623560    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 23:36:56.637214 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623563    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 23:36:56.637747 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623565    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 23:36:56.637747 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.623568    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 23:36:56.637747 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.623573    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 04 23:36:56.637747 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.631432    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 04 23:36:56.637747 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.631448    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 04 23:36:56.637747 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.631468    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 04 23:36:56.637747 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.631480    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 04 23:36:56.637747 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631609    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 23:36:56.637747 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631617    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 23:36:56.637747 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631622    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 23:36:56.637747 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631626    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 23:36:56.637747 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631630    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 23:36:56.637747 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631634    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 23:36:56.637747 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631638    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 23:36:56.637747 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631651    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631656    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631661    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631665    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631670    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631675    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631679    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631683    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631687    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631691    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631695    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631699    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631703    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631708    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631712    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631716    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631720    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631724    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631728    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631732    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 23:36:56.638124 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631736    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631742    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631746    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631750    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631754    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631759    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631763    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631767    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631771    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631774    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631778    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631782    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631786    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631798    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631802    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631807    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631811    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631815    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631819    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631823    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 23:36:56.638755 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631827    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631831    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631835    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631839    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631843    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631847    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631851    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631855    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631859    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631863    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631867    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631871    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631875    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631879    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631883    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631887    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631891    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631895    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631899    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631904    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 23:36:56.639336 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631908    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631912    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631924    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631928    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631932    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631936    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631947    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631951    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631955    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631959    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631964    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631967    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631975    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631981    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631985    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631989    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631994    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.631998    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632002    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 23:36:56.640031 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632006    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 23:36:56.640555 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632013    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 23:36:56.640555 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632019    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 23:36:56.640555 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632025    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 23:36:56.640555 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632031    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 23:36:56.640555 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632044    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 23:36:56.640555 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632049    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 23:36:56.640555 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632054    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 23:36:56.640555 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632059    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 23:36:56.640555 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632063    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 23:36:56.640555 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632068    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 23:36:56.640555 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632072    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 23:36:56.640555 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632076    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 23:36:56.640555 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.632084    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 04 23:36:56.640555 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632304    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632312    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632317    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632322    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632328    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632333    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632345    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632349    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632353    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632358    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632362    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632366    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632370    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632374    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632378    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632382    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632386    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632390    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632394    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 23:36:56.641052 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632398    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632402    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632406    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632410    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632414    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632418    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632423    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632427    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632431    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632435    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632439    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632443    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632447    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632451    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632455    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632459    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632464    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632468    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632472    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632476    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 23:36:56.641564 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632489    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632494    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632498    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632502    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632506    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632510    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632514    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632518    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632522    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632526    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632530    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632534    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632538    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632541    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632546    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632550    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632554    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632558    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632564    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 23:36:56.642080 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632570    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632575    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632593    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632598    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632602    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632606    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632610    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632614    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632618    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632622    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632626    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632630    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632634    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632638    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632650    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632655    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632661    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632666    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632671    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 23:36:56.642621 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632675    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632680    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632685    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632690    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632694    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632699    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632703    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632707    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632711    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632715    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632719    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632723    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632728    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632731    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632736    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632739    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632744    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632748    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632752    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632756    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 23:36:56.643252 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632760    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 23:36:56.643832 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:56.632764    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 23:36:56.643832 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.632772    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 04 23:36:56.643832 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.632956    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 04 23:36:56.643832 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.635799    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 04 23:36:56.643832 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.636703    2557 server.go:1019] "Starting client certificate rotation"
Sep 04 23:36:56.643832 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.636812    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 04 23:36:56.643832 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.636851    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 04 23:36:56.643832 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.641226    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 04 23:36:56.645620 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.645606    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 04 23:36:56.655388 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.655374    2557 log.go:25] "Validated CRI v1 runtime API"
Sep 04 23:36:56.658308 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.658294    2557 log.go:25] "Validated CRI v1 image API"
Sep 04 23:36:56.660068 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.660046    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 04 23:36:56.662846 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.662829    2557 fs.go:135] Filesystem UUIDs: map[436cd6e0-595f-46b5-a219-b2beb50d53dd:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d2c1444f-264d-4533-9045-117801ada063:/dev/nvme0n1p3]
Sep 04 23:36:56.662909 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.662845    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 04 23:36:56.667365 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.667351    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 04 23:36:56.668173 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.668070    2557 manager.go:216] Machine: {Timestamp:2026-09-04 23:36:56.666673592 +0000 UTC m=+0.328847325 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3080101 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec229b213068058312c5c3e084ac766e SystemUUID:ec229b21-3068-0583-12c5-c3e084ac766e BootID:ecc2f174-e77d-4337-a8b7-52d1bdc5ff9f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:59:49:50:e4:4f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:59:49:50:e4:4f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:58:9f:43:69:db Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 04 23:36:56.668173 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.668170    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 04 23:36:56.668275 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.668269    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 04 23:36:56.671622 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.671599    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 04 23:36:56.671768 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.671626    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 04 23:36:56.672406 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.672396    2557 topology_manager.go:138] "Creating topology manager with none policy"
Sep 04 23:36:56.672437 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.672408    2557 container_manager_linux.go:309] "Creating device plugin manager"
Sep 04 23:36:56.672437 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.672415    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 04 23:36:56.673074 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.673064    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 04 23:36:56.674357 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.674348    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 04 23:36:56.677509 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.677498    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 04 23:36:56.677624 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.677615    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 04 23:36:56.677654 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.677644    2557 kubelet.go:486] "Attempting to sync node with API server"
Sep 04 23:36:56.677654 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.677653    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 04 23:36:56.677712 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.677667    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 04 23:36:56.677712 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.677676    2557 kubelet.go:398] "Adding apiserver pod source"
Sep 04 23:36:56.677712 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.677687    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 04 23:36:56.678017 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.678006    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 04 23:36:56.678055 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.678026    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 04 23:36:56.680724 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.680701    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 04 23:36:56.683205 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.683190    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 04 23:36:56.683283 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.683228    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 04 23:36:56.684002 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.683989    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 04 23:36:56.684071 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.684011    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 04 23:36:56.684071 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.684019    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 04 23:36:56.684071 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.684027    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 04 23:36:56.684071 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.684044    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 04 23:36:56.684071 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.684052    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 04 23:36:56.684230 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.684210    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 04 23:36:56.684230 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.684220    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 04 23:36:56.684315 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.684231    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 04 23:36:56.684315 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.684240    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 04 23:36:56.684315 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.684252    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 04 23:36:56.684315 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.684264    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 04 23:36:56.685484 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.685472    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 04 23:36:56.685557 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.685486    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 04 23:36:56.687034 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.687007    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 04 23:36:56.687198 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.687173    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 04 23:36:56.689232 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.689217    2557 server.go:1268] "Started kubelet"
Sep 04 23:36:56.689308 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.689287    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 04 23:36:56.689352 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.689294    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 04 23:36:56.689401 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.689357    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 04 23:36:56.689919 ip-10-0-142-236 systemd[1]: Started Kubernetes Kubelet.
Sep 04 23:36:56.690061 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.689932    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cf6bb"
Sep 04 23:36:56.690644 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.690628    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 04 23:36:56.691103 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.691087    2557 server.go:310] "Adding debug handlers to kubelet server"
Sep 04 23:36:56.695184 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.695169    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-236.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 04 23:36:56.695274 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.695161    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 04 23:36:56.695994 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.695975    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 04 23:36:56.696085 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.695250    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-236.ec2.internal.18d242680d732fb2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-236.ec2.internal,UID:ip-10-0-142-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-236.ec2.internal,},FirstTimestamp:2026-09-04 23:36:56.689192882 +0000 UTC m=+0.351366615,LastTimestamp:2026-09-04 23:36:56.689192882 +0000 UTC m=+0.351366615,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-236.ec2.internal,}"
Sep 04 23:36:56.696563 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.696545    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 04 23:36:56.696651 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.696636    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Sep 04 23:36:56.697316 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.697288    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 04 23:36:56.697423 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.697320    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 04 23:36:56.697423 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.697285    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 04 23:36:56.697524 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.697430    2557 reconstruct.go:97] "Volume reconstruction finished"
Sep 04 23:36:56.697524 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.697440    2557 reconciler.go:29] "Reconciler: start to sync state"
Sep 04 23:36:56.697524 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.697450    2557 factory.go:55] Registering systemd factory
Sep 04 23:36:56.697524 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.697460    2557 factory.go:223] Registration of the systemd container factory successfully
Sep 04 23:36:56.697772 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.697644    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cf6bb"
Sep 04 23:36:56.697772 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.697709    2557 factory.go:153] Registering CRI-O factory
Sep 04 23:36:56.697772 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.697734    2557 factory.go:223] Registration of the crio container factory successfully
Sep 04 23:36:56.697913 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.697782    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 04 23:36:56.697913 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.697803    2557 factory.go:103] Registering Raw factory
Sep 04 23:36:56.697913 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.697816    2557 manager.go:1196] Started watching for new ooms in manager
Sep 04 23:36:56.698175 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.698162    2557 manager.go:349] Starting recovery of all containers
Sep 04 23:36:56.699547 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.699520    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 04 23:36:56.699660 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.699570    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 04 23:36:56.708574 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.708554    2557 manager.go:354] Recovery completed
Sep 04 23:36:56.713030 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.713014    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 23:36:56.714716 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.714700    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 23:36:56.714780 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.714729    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 23:36:56.714780 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.714739    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID"
Sep 04 23:36:56.715290 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.715274    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 04 23:36:56.715290 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.715289    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 04 23:36:56.715399 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.715306    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 04 23:36:56.715399 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.715313    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 23:36:56.716272 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.716045    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 23:36:56.716378 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.716366    2557 policy_none.go:49] "None policy: Start"
Sep 04 23:36:56.716428 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.716382    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 04 23:36:56.716428 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.716391    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 04 23:36:56.716897 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.716690    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 23:36:56.717058 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.717046    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID"
Sep 04 23:36:56.717298 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.717286    2557 policy_none.go:47] "Start"
Sep 04 23:36:56.743130 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.743116    2557 manager.go:337] "Starting Device Plugin manager"
Sep 04 23:36:56.743193 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.743151    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 04 23:36:56.743193 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.743165    2557 server.go:87] "Starting device plugin registration server"
Sep 04 23:36:56.743396 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.743382    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 04 23:36:56.743464 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.743397    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 04 23:36:56.743516 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.743474    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 04 23:36:56.743565 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.743551    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 04 23:36:56.743565 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.743560    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 04 23:36:56.744041 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.744015    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 04 23:36:56.744138 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.744073    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-236.ec2.internal\" not found"
Sep 04 23:36:56.826641 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.826606    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 04 23:36:56.827834 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.827817    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 04 23:36:56.827904 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.827873    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 04 23:36:56.827904 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.827899    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 04 23:36:56.827996 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.827942    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 04 23:36:56.830150 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.830129    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 23:36:56.843962 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.843938    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 23:36:56.844717 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.844695    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 23:36:56.844796 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.844732    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 23:36:56.844796 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.844746    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID"
Sep 04 23:36:56.844796 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.844767    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-236.ec2.internal"
Sep 04 23:36:56.852827 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.852812    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-236.ec2.internal"
Sep 04 23:36:56.852872 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.852833    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-236.ec2.internal\": node \"ip-10-0-142-236.ec2.internal\" not found"
Sep 04 23:36:56.864706 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.864644    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Sep 04 23:36:56.928359 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.928335    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"]
Sep 04 23:36:56.928446 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.928423    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 23:36:56.929682 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.929662    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 23:36:56.929748 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.929691    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 23:36:56.929748 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.929705    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID"
Sep 04 23:36:56.930993 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.930980    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 23:36:56.931124 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.931105    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Sep 04 23:36:56.931167 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.931133    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 23:36:56.931637 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.931608    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 23:36:56.931637 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.931617    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 23:36:56.931637 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.931629    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 23:36:56.931637 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.931638    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 23:36:56.931879 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.931652    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID"
Sep 04 23:36:56.931879 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.931639    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID"
Sep 04 23:36:56.933113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.933100    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"
Sep 04 23:36:56.933170 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.933121    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 23:36:56.933787 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.933762    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 23:36:56.933875 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.933792    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 23:36:56.933875 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.933805    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID"
Sep 04 23:36:56.955109 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.955085    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-236.ec2.internal\" not found" node="ip-10-0-142-236.ec2.internal"
Sep 04 23:36:56.959252 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.959237    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-236.ec2.internal\" not found" node="ip-10-0-142-236.ec2.internal"
Sep 04 23:36:56.965162 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:56.965148    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Sep 04 23:36:56.999542 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.999520    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4e1d86ad933e323b505e634664326c33-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"4e1d86ad933e323b505e634664326c33\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Sep 04 23:36:56.999636 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.999543    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e1d86ad933e323b505e634664326c33-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"4e1d86ad933e323b505e634664326c33\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Sep 04 23:36:56.999636 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:56.999558    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3170cdb077432b791941250ec565815c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" (UID: \"3170cdb077432b791941250ec565815c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"
Sep 04 23:36:57.065508 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:57.065484    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Sep 04 23:36:57.099821 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:57.099803    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4e1d86ad933e323b505e634664326c33-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"4e1d86ad933e323b505e634664326c33\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Sep 04 23:36:57.099892 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:57.099825    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e1d86ad933e323b505e634664326c33-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"4e1d86ad933e323b505e634664326c33\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Sep 04 23:36:57.099892 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:57.099842    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3170cdb077432b791941250ec565815c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" (UID: \"3170cdb077432b791941250ec565815c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"
Sep 04 23:36:57.099892 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:57.099866    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3170cdb077432b791941250ec565815c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" (UID: \"3170cdb077432b791941250ec565815c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"
Sep 04 23:36:57.100006 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:57.099890    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4e1d86ad933e323b505e634664326c33-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"4e1d86ad933e323b505e634664326c33\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Sep 04 23:36:57.100006 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:57.099903    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e1d86ad933e323b505e634664326c33-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"4e1d86ad933e323b505e634664326c33\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Sep 04 23:36:57.166097 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:57.166049    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Sep 04 23:36:57.257707 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:57.257680    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Sep 04 23:36:57.262401 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:57.262388    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"
Sep 04 23:36:57.266427 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:57.266409    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Sep 04 23:36:58.611383 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:58.611259    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Sep 04 23:36:58.711841 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:58.711784    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Sep 04 23:36:58.812345 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:58.812319    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Sep 04 23:36:58.880841 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:58.880824    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 04 23:36:58.881240 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:58.880999    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 04 23:36:58.913110 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:58.913086    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Sep 04 23:36:58.944655 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:58.944619    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-03 23:31:57 +0000 UTC" deadline="2028-02-06 11:33:45.367370649 +0000 UTC"
Sep 04 23:36:58.944655 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:58.944649    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12467h56m46.422725167s"
Sep 04 23:36:59.013720 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:59.013680    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Sep 04 23:36:59.023036 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.023017    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 23:36:59.084703 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:59.084670    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1d86ad933e323b505e634664326c33.slice/crio-6f4f880d790c30e52210e1bfa95910ff2ecfc7a46420ca689039b7d90e61af04 WatchSource:0}: Error finding container 6f4f880d790c30e52210e1bfa95910ff2ecfc7a46420ca689039b7d90e61af04: Status 404 returned error can't find the container with id 6f4f880d790c30e52210e1bfa95910ff2ecfc7a46420ca689039b7d90e61af04
Sep 04 23:36:59.090710 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.090631    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 23:36:59.092823 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:36:59.092799    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3170cdb077432b791941250ec565815c.slice/crio-145c5dffd46584eec7b7eb214a9e8f553456b939d0e6e957af62cd1918a1f444 WatchSource:0}: Error finding container 145c5dffd46584eec7b7eb214a9e8f553456b939d0e6e957af62cd1918a1f444: Status 404 returned error can't find the container with id 145c5dffd46584eec7b7eb214a9e8f553456b939d0e6e957af62cd1918a1f444
Sep 04 23:36:59.113981 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:59.113957    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Sep 04 23:36:59.214428 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:59.214393    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found"
Sep 04 23:36:59.280025 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.279951    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 23:36:59.330575 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.330550    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 23:36:59.341783 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.341755    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"
Sep 04 23:36:59.355069 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.355049    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 04 23:36:59.355959 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.355931    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"
Sep 04 23:36:59.366347 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.366331    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 04 23:36:59.723922 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.723823    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 23:36:59.922995 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.922962    2557 apiserver.go:52] "Watching apiserver"
Sep 04 23:36:59.930948 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.930912    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 04 23:36:59.931349 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.931325    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-rkpz9","kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal","openshift-multus/multus-jjnzw","openshift-network-diagnostics/network-check-target-ngp84","openshift-network-operator/iptables-alerter-42mcf","openshift-ovn-kubernetes/ovnkube-node-w6m5m","kube-system/global-pull-secret-syncer-sj4n7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9","openshift-cluster-node-tuning-operator/tuned-8mvzr","openshift-dns/node-resolver-brgnz","openshift-image-registry/node-ca-wkxtm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal","openshift-multus/multus-additional-cni-plugins-wg92b","openshift-multus/network-metrics-daemon-rd7qq"]
Sep 04 23:36:59.932882 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.932862    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:36:59.932994 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:59.932970    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:36:59.934163 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.934144    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:36:59.935571 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.935540    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.936634 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.936336    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 04 23:36:59.936634 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.936479    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 04 23:36:59.936634 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.936537    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 04 23:36:59.936847 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.936737    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 04 23:36:59.936901 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.936861    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mh9l8\""
Sep 04 23:36:59.936901 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.936877    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 04 23:36:59.937265 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.937163    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:36:59.937265 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:59.937231    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:36:59.937631 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.937611    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 04 23:36:59.937885 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.937867    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zsntb\""
Sep 04 23:36:59.938567 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.938546    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-42mcf"
Sep 04 23:36:59.940217 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.940198    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.940469 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.940452    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 04 23:36:59.940888 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.940869    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 04 23:36:59.940989 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.940885    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 04 23:36:59.940989 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.940967    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2fk7l\""
Sep 04 23:36:59.941100 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.940857    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 04 23:36:59.941814 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.941756    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:36:59.943069 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.943021    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rkpz9"
Sep 04 23:36:59.943318 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.943297    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 04 23:36:59.944386 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.944369    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 04 23:36:59.945973 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.944988    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m4rh4\""
Sep 04 23:36:59.945973 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.945122    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 04 23:36:59.945973 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.945283    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 04 23:36:59.945973 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.945302    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x74bp\""
Sep 04 23:36:59.945973 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.945316    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 04 23:36:59.945973 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.945887    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 04 23:36:59.946269 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.946155    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 04 23:36:59.947393 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.946488    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 04 23:36:59.948319 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.948300    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 04 23:36:59.948492 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.948369    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8ktjf\""
Sep 04 23:36:59.948693 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.948539    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 04 23:36:59.949189 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.949169    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:36:59.949276 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:36:59.949235    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:36:59.950734 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.950649    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-brgnz"
Sep 04 23:36:59.952501 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.952451    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wkxtm"
Sep 04 23:36:59.952892 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.952868    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-etc-kubernetes\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.953024 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.952909    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-node-log\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.953024 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.952930    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-cni-netd\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.953024 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.952967    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-sysconfig\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:36:59.953024 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953004    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-systemd\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:36:59.953231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953052    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-sys\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:36:59.953231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953081    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-cni-bin\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.953231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953104    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.953231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953149    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5105066c-f4b4-450a-944c-e7dcd6964851-ovnkube-config\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.953231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953180    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-cnibin\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.953473 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953217    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-run-openvswitch\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.953473 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953261    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-lib-modules\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:36:59.953473 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953283    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-tuned\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:36:59.953473 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953304    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9dc959dc-ab62-40f7-81ea-e62e4aafed21-dbus\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:36:59.953473 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953326    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/740e83e1-71de-4cca-ba79-614a1f20be97-cni-binary-copy\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:36:59.953473 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953347    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-run\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:36:59.953473 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953401    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.953473 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953424    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-tmp\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:36:59.953473 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953448    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e356b35c-5280-4e59-9cd2-303af05624a3-konnectivity-ca\") pod \"konnectivity-agent-rkpz9\" (UID: \"e356b35c-5280-4e59-9cd2-303af05624a3\") " pod="kube-system/konnectivity-agent-rkpz9"
Sep 04 23:36:59.953473 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953472    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-var-lib-cni-multus\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953493    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-hostroot\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953519    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-slash\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953541    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790-iptables-alerter-script\") pod \"iptables-alerter-42mcf\" (UID: \"ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790\") " pod="openshift-network-operator/iptables-alerter-42mcf"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953566    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/740e83e1-71de-4cca-ba79-614a1f20be97-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953601    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmctm\" (UniqueName: \"kubernetes.io/projected/740e83e1-71de-4cca-ba79-614a1f20be97-kube-api-access-dmctm\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953625    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-multus-socket-dir-parent\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953648    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-sysctl-d\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953669    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd119fb-c2f0-458e-819e-f2de497ef5dd-cni-binary-copy\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953692    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh9p6\" (UniqueName: \"kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6\") pod \"network-check-target-ngp84\" (UID: \"3d5cbc4c-5f2e-4611-85ee-70e51ebadd34\") " pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953715    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-var-lib-kubelet\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953737    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-host\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953760    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44pvz\" (UniqueName: \"kubernetes.io/projected/ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790-kube-api-access-44pvz\") pod \"iptables-alerter-42mcf\" (UID: \"ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790\") " pod="openshift-network-operator/iptables-alerter-42mcf"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953784    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5105066c-f4b4-450a-944c-e7dcd6964851-ovnkube-script-lib\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953810    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5105066c-f4b4-450a-944c-e7dcd6964851-env-overrides\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953835    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqvvg\" (UniqueName: \"kubernetes.io/projected/5105066c-f4b4-450a-944c-e7dcd6964851-kube-api-access-hqvvg\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.953923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953864    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/740e83e1-71de-4cca-ba79-614a1f20be97-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953895    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-multus-conf-dir\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953920    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-var-lib-openvswitch\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.953999    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-var-lib-kubelet\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954022    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-run-ovn\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954043    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/740e83e1-71de-4cca-ba79-614a1f20be97-cnibin\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954069    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-run-multus-certs\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954113    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954135    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e356b35c-5280-4e59-9cd2-303af05624a3-agent-certs\") pod \"konnectivity-agent-rkpz9\" (UID: \"e356b35c-5280-4e59-9cd2-303af05624a3\") " pod="kube-system/konnectivity-agent-rkpz9"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954286    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-system-cni-dir\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954312    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgwj4\" (UniqueName: \"kubernetes.io/projected/cfd119fb-c2f0-458e-819e-f2de497ef5dd-kube-api-access-lgwj4\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954341    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-systemd-units\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954407    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5105066c-f4b4-450a-944c-e7dcd6964851-ovn-node-metrics-cert\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954435    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-sysctl-conf\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954457    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-var-lib-cni-bin\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954464    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954490    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-kubelet\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.954672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954542    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-modprobe-d\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:36:59.955471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954567    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9dc959dc-ab62-40f7-81ea-e62e4aafed21-kubelet-config\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:36:59.955471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954591    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790-host-slash\") pod \"iptables-alerter-42mcf\" (UID: \"ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790\") " pod="openshift-network-operator/iptables-alerter-42mcf"
Sep 04 23:36:59.955471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954615    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/740e83e1-71de-4cca-ba79-614a1f20be97-system-cni-dir\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:36:59.955471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954669    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-run-k8s-cni-cncf-io\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.955471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954700    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-run-netns\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.955471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954780    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-run-systemd\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.955471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954807    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-etc-openvswitch\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.955471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954830    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/740e83e1-71de-4cca-ba79-614a1f20be97-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:36:59.955471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954857    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfd119fb-c2f0-458e-819e-f2de497ef5dd-multus-daemon-config\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.955471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954909    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-run-netns\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.955471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954950    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-log-socket\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:36:59.955471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954973    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-kubernetes\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:36:59.955471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.954999    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxhl\" (UniqueName: \"kubernetes.io/projected/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-kube-api-access-smxhl\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:36:59.955471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.955035    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-multus-cni-dir\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:36:59.956278 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.955856    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 04 23:36:59.956278 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.956194    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 04 23:36:59.956377 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.956295    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qjh8x\""
Sep 04 23:36:59.956438 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.956412    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 04 23:36:59.956506 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.956488    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 04 23:36:59.956691 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.956674    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 04 23:36:59.956748 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.956717    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 04 23:36:59.957534 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.957517    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 04 23:36:59.957696 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.957621    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-76xgw\""
Sep 04 23:36:59.957696 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.957693    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 04 23:36:59.957883 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.957867    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bfhzv\""
Sep 04 23:36:59.957953 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.957867    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 04 23:36:59.972137 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.972097    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b79nq"
Sep 04 23:36:59.982029 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:36:59.981847    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b79nq"
Sep 04 23:37:00.042395 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.042374    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 04 23:37:00.055344 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055315    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-lib-modules\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.055344 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055345    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-tuned\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.055520 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055365    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z28p\" (UniqueName: \"kubernetes.io/projected/f60b1cfd-03dc-43f3-9cce-a0bbffbb4226-kube-api-access-8z28p\") pod \"node-resolver-brgnz\" (UID: \"f60b1cfd-03dc-43f3-9cce-a0bbffbb4226\") " pod="openshift-dns/node-resolver-brgnz"
Sep 04 23:37:00.055520 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055390    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9dc959dc-ab62-40f7-81ea-e62e4aafed21-dbus\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:00.055520 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055409    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/740e83e1-71de-4cca-ba79-614a1f20be97-cni-binary-copy\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:37:00.055520 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055453    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-run\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.055520 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055490    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.055520 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055510    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-tmp\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.055758 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055534    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc24z\" (UniqueName: \"kubernetes.io/projected/0686ad4c-9eac-4756-a85d-8da561ee8e6a-kube-api-access-vc24z\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.055758 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055563    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-lib-modules\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.055758 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055588    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.055758 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055614    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-run\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.055758 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055625    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9dc959dc-ab62-40f7-81ea-e62e4aafed21-dbus\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:00.055758 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055628    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e356b35c-5280-4e59-9cd2-303af05624a3-konnectivity-ca\") pod \"konnectivity-agent-rkpz9\" (UID: \"e356b35c-5280-4e59-9cd2-303af05624a3\") " pod="kube-system/konnectivity-agent-rkpz9"
Sep 04 23:37:00.055758 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055735    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-var-lib-cni-multus\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.055758 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055757    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-hostroot\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.056099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055773    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-slash\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.056099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055811    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-slash\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.056099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055811    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-var-lib-cni-multus\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.056099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055833    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-hostroot\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.056099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055840    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d6a5b60d-4a14-49cc-8be4-f86989c0bd88-serviceca\") pod \"node-ca-wkxtm\" (UID: \"d6a5b60d-4a14-49cc-8be4-f86989c0bd88\") " pod="openshift-image-registry/node-ca-wkxtm"
Sep 04 23:37:00.056099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055870    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790-iptables-alerter-script\") pod \"iptables-alerter-42mcf\" (UID: \"ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790\") " pod="openshift-network-operator/iptables-alerter-42mcf"
Sep 04 23:37:00.056099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055890    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 04 23:37:00.056099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055897    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/740e83e1-71de-4cca-ba79-614a1f20be97-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:37:00.056099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.055998    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmctm\" (UniqueName: \"kubernetes.io/projected/740e83e1-71de-4cca-ba79-614a1f20be97-kube-api-access-dmctm\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:37:00.056099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056050    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-multus-socket-dir-parent\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.056099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056078    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-sysctl-d\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056119    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6a5b60d-4a14-49cc-8be4-f86989c0bd88-host\") pod \"node-ca-wkxtm\" (UID: \"d6a5b60d-4a14-49cc-8be4-f86989c0bd88\") " pod="openshift-image-registry/node-ca-wkxtm"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056143    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd119fb-c2f0-458e-819e-f2de497ef5dd-cni-binary-copy\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056162    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e356b35c-5280-4e59-9cd2-303af05624a3-konnectivity-ca\") pod \"konnectivity-agent-rkpz9\" (UID: \"e356b35c-5280-4e59-9cd2-303af05624a3\") " pod="kube-system/konnectivity-agent-rkpz9"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056167    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9p6\" (UniqueName: \"kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6\") pod \"network-check-target-ngp84\" (UID: \"3d5cbc4c-5f2e-4611-85ee-70e51ebadd34\") " pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056203    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-var-lib-kubelet\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056224    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-host\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056251    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056256    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-sysctl-d\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056278    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44pvz\" (UniqueName: \"kubernetes.io/projected/ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790-kube-api-access-44pvz\") pod \"iptables-alerter-42mcf\" (UID: \"ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790\") " pod="openshift-network-operator/iptables-alerter-42mcf"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056304    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5105066c-f4b4-450a-944c-e7dcd6964851-ovnkube-script-lib\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056326    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-socket-dir\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056350    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-registration-dir\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056373    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5105066c-f4b4-450a-944c-e7dcd6964851-env-overrides\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056420    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqvvg\" (UniqueName: \"kubernetes.io/projected/5105066c-f4b4-450a-944c-e7dcd6964851-kube-api-access-hqvvg\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056446    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/740e83e1-71de-4cca-ba79-614a1f20be97-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056448    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/740e83e1-71de-4cca-ba79-614a1f20be97-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:37:00.056623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056463    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790-iptables-alerter-script\") pod \"iptables-alerter-42mcf\" (UID: \"ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790\") " pod="openshift-network-operator/iptables-alerter-42mcf"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056470    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-multus-conf-dir\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056494    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-var-lib-openvswitch\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056521    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt594\" (UniqueName: \"kubernetes.io/projected/ca3463da-4922-4b81-a71e-8769f4a90494-kube-api-access-xt594\") pod \"network-metrics-daemon-rd7qq\" (UID: \"ca3463da-4922-4b81-a71e-8769f4a90494\") " pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056529    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-multus-socket-dir-parent\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056162    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/740e83e1-71de-4cca-ba79-614a1f20be97-cni-binary-copy\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056546    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-var-lib-kubelet\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056590    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-var-lib-kubelet\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056602    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-host\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056617    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-run-ovn\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056644    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60b1cfd-03dc-43f3-9cce-a0bbffbb4226-tmp-dir\") pod \"node-resolver-brgnz\" (UID: \"f60b1cfd-03dc-43f3-9cce-a0bbffbb4226\") " pod="openshift-dns/node-resolver-brgnz"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056670    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/740e83e1-71de-4cca-ba79-614a1f20be97-cnibin\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056696    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-run-multus-certs\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056720    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056747    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056770    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e356b35c-5280-4e59-9cd2-303af05624a3-agent-certs\") pod \"konnectivity-agent-rkpz9\" (UID: \"e356b35c-5280-4e59-9cd2-303af05624a3\") " pod="kube-system/konnectivity-agent-rkpz9"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056792    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-system-cni-dir\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.057432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056824    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwj4\" (UniqueName: \"kubernetes.io/projected/cfd119fb-c2f0-458e-819e-f2de497ef5dd-kube-api-access-lgwj4\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056846    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-systemd-units\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056868    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5105066c-f4b4-450a-944c-e7dcd6964851-ovn-node-metrics-cert\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056895    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-sysctl-conf\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056920    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-var-lib-cni-bin\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056963    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-kubelet\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056992    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-modprobe-d\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057010    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5105066c-f4b4-450a-944c-e7dcd6964851-env-overrides\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057017    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f60b1cfd-03dc-43f3-9cce-a0bbffbb4226-hosts-file\") pod \"node-resolver-brgnz\" (UID: \"f60b1cfd-03dc-43f3-9cce-a0bbffbb4226\") " pod="openshift-dns/node-resolver-brgnz"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057019    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd119fb-c2f0-458e-819e-f2de497ef5dd-cni-binary-copy\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057044    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-592mf\" (UniqueName: \"kubernetes.io/projected/d6a5b60d-4a14-49cc-8be4-f86989c0bd88-kube-api-access-592mf\") pod \"node-ca-wkxtm\" (UID: \"d6a5b60d-4a14-49cc-8be4-f86989c0bd88\") " pod="openshift-image-registry/node-ca-wkxtm"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057070    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-run-multus-certs\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057070    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-device-dir\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057113    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9dc959dc-ab62-40f7-81ea-e62e4aafed21-kubelet-config\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057127    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-var-lib-openvswitch\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057156    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5105066c-f4b4-450a-944c-e7dcd6964851-ovnkube-script-lib\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057168    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-multus-conf-dir\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.057174    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:00.058251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057182    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9dc959dc-ab62-40f7-81ea-e62e4aafed21-kubelet-config\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.056644    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-var-lib-kubelet\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057194    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790-host-slash\") pod \"iptables-alerter-42mcf\" (UID: \"ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790\") " pod="openshift-network-operator/iptables-alerter-42mcf"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057082    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-run-ovn\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057228    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/740e83e1-71de-4cca-ba79-614a1f20be97-system-cni-dir\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.057247    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret podName:9dc959dc-ab62-40f7-81ea-e62e4aafed21 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:00.5572261 +0000 UTC m=+2.975044410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret") pod "global-pull-secret-syncer-sj4n7" (UID: "9dc959dc-ab62-40f7-81ea-e62e4aafed21") : object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057283    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-system-cni-dir\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057349    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-modprobe-d\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057399    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-sysctl-conf\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057457    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/740e83e1-71de-4cca-ba79-614a1f20be97-cnibin\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057495    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-systemd-units\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057564    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/740e83e1-71de-4cca-ba79-614a1f20be97-system-cni-dir\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057585    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-run-k8s-cni-cncf-io\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057625    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-run-netns\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057634    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/740e83e1-71de-4cca-ba79-614a1f20be97-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057660    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-run-systemd\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057674    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790-host-slash\") pod \"iptables-alerter-42mcf\" (UID: \"ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790\") " pod="openshift-network-operator/iptables-alerter-42mcf"
Sep 04 23:37:00.059113 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057681    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-run-k8s-cni-cncf-io\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057717    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-etc-openvswitch\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057837    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-kubelet\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057738    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-run-netns\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057762    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-etc-openvswitch\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057730    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-run-systemd\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057836    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/740e83e1-71de-4cca-ba79-614a1f20be97-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057891    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfd119fb-c2f0-458e-819e-f2de497ef5dd-multus-daemon-config\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057790    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-host-var-lib-cni-bin\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.057966    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-run-netns\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058005    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-run-netns\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058011    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-log-socket\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058038    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-kubernetes\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058023    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/740e83e1-71de-4cca-ba79-614a1f20be97-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058075    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-log-socket\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058094    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxhl\" (UniqueName: \"kubernetes.io/projected/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-kube-api-access-smxhl\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058111    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-kubernetes\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058120    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-multus-cni-dir\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.059823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058145    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-etc-kubernetes\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058166    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-node-log\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058189    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-cni-netd\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058211    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-sysconfig\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058234    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-systemd\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058254    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-sys\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058260    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-node-log\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058278    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-sys-fs\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058302    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-cni-bin\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058320    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-multus-cni-dir\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058327    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058354    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5105066c-f4b4-450a-944c-e7dcd6964851-ovnkube-config\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058358    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-etc-kubernetes\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058394    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfd119fb-c2f0-458e-819e-f2de497ef5dd-multus-daemon-config\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058405    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs\") pod \"network-metrics-daemon-rd7qq\" (UID: \"ca3463da-4922-4b81-a71e-8769f4a90494\") " pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058422    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-sys\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058433    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-cnibin\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058457    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-run-openvswitch\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.060590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058459    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-cni-netd\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.061238 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058496    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-sysconfig\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.061238 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058539    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-systemd\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.061238 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058542    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-cni-bin\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.061238 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058583    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-run-openvswitch\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.061238 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058584    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5105066c-f4b4-450a-944c-e7dcd6964851-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.061238 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.058631    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd119fb-c2f0-458e-819e-f2de497ef5dd-cnibin\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.061238 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.059031    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5105066c-f4b4-450a-944c-e7dcd6964851-ovnkube-config\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.061238 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.060220    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-etc-tuned\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.061238 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.060326    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5105066c-f4b4-450a-944c-e7dcd6964851-ovn-node-metrics-cert\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.061238 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.060489    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-tmp\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.061238 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.061201    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e356b35c-5280-4e59-9cd2-303af05624a3-agent-certs\") pod \"konnectivity-agent-rkpz9\" (UID: \"e356b35c-5280-4e59-9cd2-303af05624a3\") " pod="kube-system/konnectivity-agent-rkpz9"
Sep 04 23:37:00.063544 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.063525    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 23:37:00.063653 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.063549    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 23:37:00.063653 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.063560    2557 projected.go:196] Error preparing data for projected volume kube-api-access-gh9p6 for pod openshift-network-diagnostics/network-check-target-ngp84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:37:00.063653 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.063616    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6 podName:3d5cbc4c-5f2e-4611-85ee-70e51ebadd34 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:00.563602544 +0000 UTC m=+2.981420840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gh9p6" (UniqueName: "kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6") pod "network-check-target-ngp84" (UID: "3d5cbc4c-5f2e-4611-85ee-70e51ebadd34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:37:00.066219 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.066196    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44pvz\" (UniqueName: \"kubernetes.io/projected/ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790-kube-api-access-44pvz\") pod \"iptables-alerter-42mcf\" (UID: \"ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790\") " pod="openshift-network-operator/iptables-alerter-42mcf"
Sep 04 23:37:00.066299 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.066229    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmctm\" (UniqueName: \"kubernetes.io/projected/740e83e1-71de-4cca-ba79-614a1f20be97-kube-api-access-dmctm\") pod \"multus-additional-cni-plugins-wg92b\" (UID: \"740e83e1-71de-4cca-ba79-614a1f20be97\") " pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:37:00.066468 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.066453    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxhl\" (UniqueName: \"kubernetes.io/projected/fcc26e56-e58d-4030-9b3e-d87b573d6a8c-kube-api-access-smxhl\") pod \"tuned-8mvzr\" (UID: \"fcc26e56-e58d-4030-9b3e-d87b573d6a8c\") " pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.066902 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.066878    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqvvg\" (UniqueName: \"kubernetes.io/projected/5105066c-f4b4-450a-944c-e7dcd6964851-kube-api-access-hqvvg\") pod \"ovnkube-node-w6m5m\" (UID: \"5105066c-f4b4-450a-944c-e7dcd6964851\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.067437 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.067408    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwj4\" (UniqueName: \"kubernetes.io/projected/cfd119fb-c2f0-458e-819e-f2de497ef5dd-kube-api-access-lgwj4\") pod \"multus-jjnzw\" (UID: \"cfd119fb-c2f0-458e-819e-f2de497ef5dd\") " pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.077590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.077540    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" event={"ID":"3170cdb077432b791941250ec565815c","Type":"ContainerStarted","Data":"145c5dffd46584eec7b7eb214a9e8f553456b939d0e6e957af62cd1918a1f444"}
Sep 04 23:37:00.078852 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.078828    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" event={"ID":"4e1d86ad933e323b505e634664326c33","Type":"ContainerStarted","Data":"6f4f880d790c30e52210e1bfa95910ff2ecfc7a46420ca689039b7d90e61af04"}
Sep 04 23:37:00.159230 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159198    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60b1cfd-03dc-43f3-9cce-a0bbffbb4226-tmp-dir\") pod \"node-resolver-brgnz\" (UID: \"f60b1cfd-03dc-43f3-9cce-a0bbffbb4226\") " pod="openshift-dns/node-resolver-brgnz"
Sep 04 23:37:00.159407 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159244    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.159407 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159288    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f60b1cfd-03dc-43f3-9cce-a0bbffbb4226-hosts-file\") pod \"node-resolver-brgnz\" (UID: \"f60b1cfd-03dc-43f3-9cce-a0bbffbb4226\") " pod="openshift-dns/node-resolver-brgnz"
Sep 04 23:37:00.159407 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159313    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-592mf\" (UniqueName: \"kubernetes.io/projected/d6a5b60d-4a14-49cc-8be4-f86989c0bd88-kube-api-access-592mf\") pod \"node-ca-wkxtm\" (UID: \"d6a5b60d-4a14-49cc-8be4-f86989c0bd88\") " pod="openshift-image-registry/node-ca-wkxtm"
Sep 04 23:37:00.159407 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159338    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-device-dir\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.159407 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159375    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-sys-fs\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.159407 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159401    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs\") pod \"network-metrics-daemon-rd7qq\" (UID: \"ca3463da-4922-4b81-a71e-8769f4a90494\") " pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:00.159704 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159428    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z28p\" (UniqueName: \"kubernetes.io/projected/f60b1cfd-03dc-43f3-9cce-a0bbffbb4226-kube-api-access-8z28p\") pod \"node-resolver-brgnz\" (UID: \"f60b1cfd-03dc-43f3-9cce-a0bbffbb4226\") " pod="openshift-dns/node-resolver-brgnz"
Sep 04 23:37:00.159704 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159457    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc24z\" (UniqueName: \"kubernetes.io/projected/0686ad4c-9eac-4756-a85d-8da561ee8e6a-kube-api-access-vc24z\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.159704 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159466    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.159704 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159484    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d6a5b60d-4a14-49cc-8be4-f86989c0bd88-serviceca\") pod \"node-ca-wkxtm\" (UID: \"d6a5b60d-4a14-49cc-8be4-f86989c0bd88\") " pod="openshift-image-registry/node-ca-wkxtm"
Sep 04 23:37:00.159704 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159511    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6a5b60d-4a14-49cc-8be4-f86989c0bd88-host\") pod \"node-ca-wkxtm\" (UID: \"d6a5b60d-4a14-49cc-8be4-f86989c0bd88\") " pod="openshift-image-registry/node-ca-wkxtm"
Sep 04 23:37:00.159704 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.159541    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:00.159704 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159546    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.159704 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159572    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-socket-dir\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.159704 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159587    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-device-dir\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.159704 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159588    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60b1cfd-03dc-43f3-9cce-a0bbffbb4226-tmp-dir\") pod \"node-resolver-brgnz\" (UID: \"f60b1cfd-03dc-43f3-9cce-a0bbffbb4226\") " pod="openshift-dns/node-resolver-brgnz"
Sep 04 23:37:00.159704 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.159598    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs podName:ca3463da-4922-4b81-a71e-8769f4a90494 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:00.659579693 +0000 UTC m=+3.077398005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs") pod "network-metrics-daemon-rd7qq" (UID: "ca3463da-4922-4b81-a71e-8769f4a90494") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:00.159704 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159662    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-registration-dir\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.159704 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159691    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt594\" (UniqueName: \"kubernetes.io/projected/ca3463da-4922-4b81-a71e-8769f4a90494-kube-api-access-xt594\") pod \"network-metrics-daemon-rd7qq\" (UID: \"ca3463da-4922-4b81-a71e-8769f4a90494\") " pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:00.159704 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159697    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-socket-dir\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.160251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159760    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-sys-fs\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.160251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159428    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f60b1cfd-03dc-43f3-9cce-a0bbffbb4226-hosts-file\") pod \"node-resolver-brgnz\" (UID: \"f60b1cfd-03dc-43f3-9cce-a0bbffbb4226\") " pod="openshift-dns/node-resolver-brgnz"
Sep 04 23:37:00.160251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159852    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6a5b60d-4a14-49cc-8be4-f86989c0bd88-host\") pod \"node-ca-wkxtm\" (UID: \"d6a5b60d-4a14-49cc-8be4-f86989c0bd88\") " pod="openshift-image-registry/node-ca-wkxtm"
Sep 04 23:37:00.160251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159891    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-registration-dir\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.160251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.159991    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0686ad4c-9eac-4756-a85d-8da561ee8e6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.160251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.160126    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d6a5b60d-4a14-49cc-8be4-f86989c0bd88-serviceca\") pod \"node-ca-wkxtm\" (UID: \"d6a5b60d-4a14-49cc-8be4-f86989c0bd88\") " pod="openshift-image-registry/node-ca-wkxtm"
Sep 04 23:37:00.170141 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.170116    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-592mf\" (UniqueName: \"kubernetes.io/projected/d6a5b60d-4a14-49cc-8be4-f86989c0bd88-kube-api-access-592mf\") pod \"node-ca-wkxtm\" (UID: \"d6a5b60d-4a14-49cc-8be4-f86989c0bd88\") " pod="openshift-image-registry/node-ca-wkxtm"
Sep 04 23:37:00.170900 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.170872    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z28p\" (UniqueName: \"kubernetes.io/projected/f60b1cfd-03dc-43f3-9cce-a0bbffbb4226-kube-api-access-8z28p\") pod \"node-resolver-brgnz\" (UID: \"f60b1cfd-03dc-43f3-9cce-a0bbffbb4226\") " pod="openshift-dns/node-resolver-brgnz"
Sep 04 23:37:00.171002 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.170976    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc24z\" (UniqueName: \"kubernetes.io/projected/0686ad4c-9eac-4756-a85d-8da561ee8e6a-kube-api-access-vc24z\") pod \"aws-ebs-csi-driver-node-fpvr9\" (UID: \"0686ad4c-9eac-4756-a85d-8da561ee8e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.171061 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.171006    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt594\" (UniqueName: \"kubernetes.io/projected/ca3463da-4922-4b81-a71e-8769f4a90494-kube-api-access-xt594\") pod \"network-metrics-daemon-rd7qq\" (UID: \"ca3463da-4922-4b81-a71e-8769f4a90494\") " pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:00.254922 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.254860    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wg92b"
Sep 04 23:37:00.260294 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.260275    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jjnzw"
Sep 04 23:37:00.270261 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.270244    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-42mcf"
Sep 04 23:37:00.276114 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.276090    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:00.283832 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.283811    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8mvzr"
Sep 04 23:37:00.291718 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.291699    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rkpz9"
Sep 04 23:37:00.300333 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.300317    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-brgnz"
Sep 04 23:37:00.305890 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.305872    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wkxtm"
Sep 04 23:37:00.311457 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.311441    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9"
Sep 04 23:37:00.561807 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.561730    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:00.561984 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.561889    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:00.561984 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.561974    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret podName:9dc959dc-ab62-40f7-81ea-e62e4aafed21 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:01.561955297 +0000 UTC m=+3.979773592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret") pod "global-pull-secret-syncer-sj4n7" (UID: "9dc959dc-ab62-40f7-81ea-e62e4aafed21") : object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:00.662665 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.662636    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs\") pod \"network-metrics-daemon-rd7qq\" (UID: \"ca3463da-4922-4b81-a71e-8769f4a90494\") " pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:00.662824 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.662685    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9p6\" (UniqueName: \"kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6\") pod \"network-check-target-ngp84\" (UID: \"3d5cbc4c-5f2e-4611-85ee-70e51ebadd34\") " pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:00.662824 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.662795    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:00.662960 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.662868    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs podName:ca3463da-4922-4b81-a71e-8769f4a90494 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:01.662848613 +0000 UTC m=+4.080666930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs") pod "network-metrics-daemon-rd7qq" (UID: "ca3463da-4922-4b81-a71e-8769f4a90494") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:00.662960 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.662800    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 23:37:00.662960 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.662924    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 23:37:00.662960 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.662951    2557 projected.go:196] Error preparing data for projected volume kube-api-access-gh9p6 for pod openshift-network-diagnostics/network-check-target-ngp84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:37:00.663157 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:00.663002    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6 podName:3d5cbc4c-5f2e-4611-85ee-70e51ebadd34 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:01.66298622 +0000 UTC m=+4.080804520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gh9p6" (UniqueName: "kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6") pod "network-check-target-ngp84" (UID: "3d5cbc4c-5f2e-4611-85ee-70e51ebadd34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:37:00.925819 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:37:00.925658    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5105066c_f4b4_450a_944c_e7dcd6964851.slice/crio-85415ae289347a57557acaca66324449a05a2673c642a2c79e4cda666cdf435c WatchSource:0}: Error finding container 85415ae289347a57557acaca66324449a05a2673c642a2c79e4cda666cdf435c: Status 404 returned error can't find the container with id 85415ae289347a57557acaca66324449a05a2673c642a2c79e4cda666cdf435c
Sep 04 23:37:00.931982 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:37:00.931952    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740e83e1_71de_4cca_ba79_614a1f20be97.slice/crio-b5121d9c28c151e08ea94869c0fd9e51a2e2c7a3215f143bd2c082c82fb3f40b WatchSource:0}: Error finding container b5121d9c28c151e08ea94869c0fd9e51a2e2c7a3215f143bd2c082c82fb3f40b: Status 404 returned error can't find the container with id b5121d9c28c151e08ea94869c0fd9e51a2e2c7a3215f143bd2c082c82fb3f40b
Sep 04 23:37:00.933162 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:37:00.933139    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccabb3bd_e9c8_4b72_8bcc_9c7a15cb6790.slice/crio-0f63c7694d7e6f853f56f69ecd3798d49e11e9fd8178174d15e9d625a9e14d6d WatchSource:0}: Error finding container 0f63c7694d7e6f853f56f69ecd3798d49e11e9fd8178174d15e9d625a9e14d6d: Status 404 returned error can't find the container with id 0f63c7694d7e6f853f56f69ecd3798d49e11e9fd8178174d15e9d625a9e14d6d
Sep 04 23:37:00.933838 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:37:00.933784    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a5b60d_4a14_49cc_8be4_f86989c0bd88.slice/crio-f8877062202f1527cc896152e89e384d0dc7a4ec5a4376539ef23d1d8c4e2e49 WatchSource:0}: Error finding container f8877062202f1527cc896152e89e384d0dc7a4ec5a4376539ef23d1d8c4e2e49: Status 404 returned error can't find the container with id f8877062202f1527cc896152e89e384d0dc7a4ec5a4376539ef23d1d8c4e2e49
Sep 04 23:37:00.934592 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:37:00.934445    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd119fb_c2f0_458e_819e_f2de497ef5dd.slice/crio-cf8a12edbf44faabf4180debea66a3c63faf417c956af68368fad6f3a1a5521c WatchSource:0}: Error finding container cf8a12edbf44faabf4180debea66a3c63faf417c956af68368fad6f3a1a5521c: Status 404 returned error can't find the container with id cf8a12edbf44faabf4180debea66a3c63faf417c956af68368fad6f3a1a5521c
Sep 04 23:37:00.935746 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:37:00.935722    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0686ad4c_9eac_4756_a85d_8da561ee8e6a.slice/crio-255bc8554433089c0c26ac64716381ef4d6b68c1c410b5cc87c968a83ca97d1b WatchSource:0}: Error finding container 255bc8554433089c0c26ac64716381ef4d6b68c1c410b5cc87c968a83ca97d1b: Status 404 returned error can't find the container with id 255bc8554433089c0c26ac64716381ef4d6b68c1c410b5cc87c968a83ca97d1b
Sep 04 23:37:00.938154 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:37:00.938125    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode356b35c_5280_4e59_9cd2_303af05624a3.slice/crio-b77a3e591bef16de8ac133c037bee47cf40404f33fdc5cefbaaa8d78300f6453 WatchSource:0}: Error finding container b77a3e591bef16de8ac133c037bee47cf40404f33fdc5cefbaaa8d78300f6453: Status 404 returned error can't find the container with id b77a3e591bef16de8ac133c037bee47cf40404f33fdc5cefbaaa8d78300f6453
Sep 04 23:37:00.940081 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:37:00.939994    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc26e56_e58d_4030_9b3e_d87b573d6a8c.slice/crio-66c616688182458e770f925c9792d162fa48930e0506a3a2c462be8474bc88a8 WatchSource:0}: Error finding container 66c616688182458e770f925c9792d162fa48930e0506a3a2c462be8474bc88a8: Status 404 returned error can't find the container with id 66c616688182458e770f925c9792d162fa48930e0506a3a2c462be8474bc88a8
Sep 04 23:37:00.941046 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:37:00.940930    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60b1cfd_03dc_43f3_9cce_a0bbffbb4226.slice/crio-e3631238c91f35c6a286c0dc64e9711d32e95af1896a60785fe300e0437ba1f9 WatchSource:0}: Error finding container e3631238c91f35c6a286c0dc64e9711d32e95af1896a60785fe300e0437ba1f9: Status 404 returned error can't find the container with id e3631238c91f35c6a286c0dc64e9711d32e95af1896a60785fe300e0437ba1f9
Sep 04 23:37:00.983863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.983804    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-03 23:31:59 +0000 UTC" deadline="2028-04-30 11:51:50.491216033 +0000 UTC"
Sep 04 23:37:00.983863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:00.983862    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14484h14m49.507357949s"
Sep 04 23:37:01.073030 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.073005    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:01.073150 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:01.073127    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:01.082155 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.082091    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-brgnz" event={"ID":"f60b1cfd-03dc-43f3-9cce-a0bbffbb4226","Type":"ContainerStarted","Data":"e3631238c91f35c6a286c0dc64e9711d32e95af1896a60785fe300e0437ba1f9"}
Sep 04 23:37:01.083760 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.083732    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jjnzw" event={"ID":"cfd119fb-c2f0-458e-819e-f2de497ef5dd","Type":"ContainerStarted","Data":"cf8a12edbf44faabf4180debea66a3c63faf417c956af68368fad6f3a1a5521c"}
Sep 04 23:37:01.085382 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.085363    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-42mcf" event={"ID":"ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790","Type":"ContainerStarted","Data":"0f63c7694d7e6f853f56f69ecd3798d49e11e9fd8178174d15e9d625a9e14d6d"}
Sep 04 23:37:01.086493 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.086458    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8mvzr" event={"ID":"fcc26e56-e58d-4030-9b3e-d87b573d6a8c","Type":"ContainerStarted","Data":"66c616688182458e770f925c9792d162fa48930e0506a3a2c462be8474bc88a8"}
Sep 04 23:37:01.087858 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.087835    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rkpz9" event={"ID":"e356b35c-5280-4e59-9cd2-303af05624a3","Type":"ContainerStarted","Data":"b77a3e591bef16de8ac133c037bee47cf40404f33fdc5cefbaaa8d78300f6453"}
Sep 04 23:37:01.089043 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.089023    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9" event={"ID":"0686ad4c-9eac-4756-a85d-8da561ee8e6a","Type":"ContainerStarted","Data":"255bc8554433089c0c26ac64716381ef4d6b68c1c410b5cc87c968a83ca97d1b"}
Sep 04 23:37:01.090099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.090077    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wkxtm" event={"ID":"d6a5b60d-4a14-49cc-8be4-f86989c0bd88","Type":"ContainerStarted","Data":"f8877062202f1527cc896152e89e384d0dc7a4ec5a4376539ef23d1d8c4e2e49"}
Sep 04 23:37:01.091270 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.091250    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg92b" event={"ID":"740e83e1-71de-4cca-ba79-614a1f20be97","Type":"ContainerStarted","Data":"b5121d9c28c151e08ea94869c0fd9e51a2e2c7a3215f143bd2c082c82fb3f40b"}
Sep 04 23:37:01.092333 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.092310    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m" event={"ID":"5105066c-f4b4-450a-944c-e7dcd6964851","Type":"ContainerStarted","Data":"85415ae289347a57557acaca66324449a05a2673c642a2c79e4cda666cdf435c"}
Sep 04 23:37:01.094340 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.094321    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" event={"ID":"3170cdb077432b791941250ec565815c","Type":"ContainerStarted","Data":"9da9e9b8660d52389517273afb6a44179324942b60d23686c0f101713ecb10f3"}
Sep 04 23:37:01.568987 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.568658    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:01.568987 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:01.568969    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:01.569200 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:01.569046    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret podName:9dc959dc-ab62-40f7-81ea-e62e4aafed21 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:03.569027756 +0000 UTC m=+5.986846070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret") pod "global-pull-secret-syncer-sj4n7" (UID: "9dc959dc-ab62-40f7-81ea-e62e4aafed21") : object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:01.670176 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.669402    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs\") pod \"network-metrics-daemon-rd7qq\" (UID: \"ca3463da-4922-4b81-a71e-8769f4a90494\") " pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:01.670176 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.669460    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9p6\" (UniqueName: \"kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6\") pod \"network-check-target-ngp84\" (UID: \"3d5cbc4c-5f2e-4611-85ee-70e51ebadd34\") " pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:01.670176 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:01.669611    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 23:37:01.670176 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:01.669628    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 23:37:01.670176 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:01.669639    2557 projected.go:196] Error preparing data for projected volume kube-api-access-gh9p6 for pod openshift-network-diagnostics/network-check-target-ngp84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:37:01.670176 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:01.669692    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6 podName:3d5cbc4c-5f2e-4611-85ee-70e51ebadd34 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:03.669674176 +0000 UTC m=+6.087492492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gh9p6" (UniqueName: "kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6") pod "network-check-target-ngp84" (UID: "3d5cbc4c-5f2e-4611-85ee-70e51ebadd34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:37:01.670176 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:01.670095    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:01.670176 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:01.670144    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs podName:ca3463da-4922-4b81-a71e-8769f4a90494 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:03.670129383 +0000 UTC m=+6.087947684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs") pod "network-metrics-daemon-rd7qq" (UID: "ca3463da-4922-4b81-a71e-8769f4a90494") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:01.984419 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.984321    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-03 23:31:59 +0000 UTC" deadline="2028-03-26 21:10:11.152330138 +0000 UTC"
Sep 04 23:37:01.984419 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:01.984365    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13653h33m9.167970159s"
Sep 04 23:37:02.073130 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:02.073100    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:02.073291 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:02.073222    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:02.073752 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:02.073730    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:02.073865 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:02.073836    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:02.124991 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:02.123854    2557 generic.go:356] "Generic (PLEG): container finished" podID="4e1d86ad933e323b505e634664326c33" containerID="914407d30fd0b02317f772232723d3e64acf801c173e41fe38312d0d5ccc2263" exitCode=0
Sep 04 23:37:02.124991 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:02.124757    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" event={"ID":"4e1d86ad933e323b505e634664326c33","Type":"ContainerDied","Data":"914407d30fd0b02317f772232723d3e64acf801c173e41fe38312d0d5ccc2263"}
Sep 04 23:37:02.139315 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:02.138379    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" podStartSLOduration=3.138361332 podStartE2EDuration="3.138361332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:01.106889856 +0000 UTC m=+3.524708173" watchObservedRunningTime="2026-09-04 23:37:02.138361332 +0000 UTC m=+4.556179651"
Sep 04 23:37:03.073768 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:03.073274    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:03.073768 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:03.073403    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:03.136968 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:03.136529    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" event={"ID":"4e1d86ad933e323b505e634664326c33","Type":"ContainerStarted","Data":"e690105faeabb73f49535f57daa748265f9c0d7db31534f10e8dff05ca4caaf3"}
Sep 04 23:37:03.585906 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:03.585871    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:03.586090 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:03.586075    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:03.586161 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:03.586133    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret podName:9dc959dc-ab62-40f7-81ea-e62e4aafed21 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:07.586115713 +0000 UTC m=+10.003934023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret") pod "global-pull-secret-syncer-sj4n7" (UID: "9dc959dc-ab62-40f7-81ea-e62e4aafed21") : object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:03.686794 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:03.686764    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs\") pod \"network-metrics-daemon-rd7qq\" (UID: \"ca3463da-4922-4b81-a71e-8769f4a90494\") " pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:03.686980 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:03.686818    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9p6\" (UniqueName: \"kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6\") pod \"network-check-target-ngp84\" (UID: \"3d5cbc4c-5f2e-4611-85ee-70e51ebadd34\") " pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:03.686980 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:03.686964    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 23:37:03.687103 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:03.686982    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 23:37:03.687103 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:03.686996    2557 projected.go:196] Error preparing data for projected volume kube-api-access-gh9p6 for pod openshift-network-diagnostics/network-check-target-ngp84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:37:03.687103 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:03.687047    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6 podName:3d5cbc4c-5f2e-4611-85ee-70e51ebadd34 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:07.687030322 +0000 UTC m=+10.104848632 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gh9p6" (UniqueName: "kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6") pod "network-check-target-ngp84" (UID: "3d5cbc4c-5f2e-4611-85ee-70e51ebadd34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:37:03.687431 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:03.687410    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:03.687512 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:03.687465    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs podName:ca3463da-4922-4b81-a71e-8769f4a90494 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:07.687450195 +0000 UTC m=+10.105268506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs") pod "network-metrics-daemon-rd7qq" (UID: "ca3463da-4922-4b81-a71e-8769f4a90494") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:04.073334 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:04.073263    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:04.073491 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:04.073406    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:04.073798 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:04.073776    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:04.074191 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:04.073871    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:05.073531 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:05.073335    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:05.073531 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:05.073487    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:06.075368 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:06.075335    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:06.075811 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:06.075463    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:06.075811 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:06.075669    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:06.075811 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:06.075757    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:07.073153 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:07.073075    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:07.073307 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:07.073223    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:07.621622 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:07.621584    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:07.622096 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:07.621780    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:07.622096 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:07.621845    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret podName:9dc959dc-ab62-40f7-81ea-e62e4aafed21 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:15.621826775 +0000 UTC m=+18.039645074 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret") pod "global-pull-secret-syncer-sj4n7" (UID: "9dc959dc-ab62-40f7-81ea-e62e4aafed21") : object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:07.723017 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:07.722977    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9p6\" (UniqueName: \"kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6\") pod \"network-check-target-ngp84\" (UID: \"3d5cbc4c-5f2e-4611-85ee-70e51ebadd34\") " pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:07.723179 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:07.723057    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs\") pod \"network-metrics-daemon-rd7qq\" (UID: \"ca3463da-4922-4b81-a71e-8769f4a90494\") " pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:07.723304 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:07.723178    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:07.723304 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:07.723236    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs podName:ca3463da-4922-4b81-a71e-8769f4a90494 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:15.723221182 +0000 UTC m=+18.141039500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs") pod "network-metrics-daemon-rd7qq" (UID: "ca3463da-4922-4b81-a71e-8769f4a90494") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:07.723304 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:07.723176    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 23:37:07.723304 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:07.723279    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 23:37:07.723304 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:07.723292    2557 projected.go:196] Error preparing data for projected volume kube-api-access-gh9p6 for pod openshift-network-diagnostics/network-check-target-ngp84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:37:07.723554 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:07.723330    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6 podName:3d5cbc4c-5f2e-4611-85ee-70e51ebadd34 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:15.723318167 +0000 UTC m=+18.141136468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gh9p6" (UniqueName: "kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6") pod "network-check-target-ngp84" (UID: "3d5cbc4c-5f2e-4611-85ee-70e51ebadd34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:37:08.074346 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:08.074244    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:08.074491 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:08.074350    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:08.074887 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:08.074672    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:08.074887 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:08.074805    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:09.073221 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:09.073110    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:09.073664 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:09.073270    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:10.073126 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:10.073091    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:10.073283 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:10.073091    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:10.073283 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:10.073213    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:10.073283 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:10.073257    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:11.072552 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:11.072528    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:11.072650 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:11.072626    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:12.072819 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:12.072785    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:12.073291 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:12.072788    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:12.073291 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:12.072901    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:12.073291 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:12.073013    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:13.073313 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:13.073213    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:13.073748 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:13.073346    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:14.072552 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:14.072519    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:14.072699 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:14.072639    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:14.072699 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:14.072681    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:14.072788 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:14.072751    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:15.072509 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:15.072473    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:15.072972 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:15.072620    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:15.681860 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:15.681767    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:15.682061 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:15.681904    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:15.682061 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:15.681994    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret podName:9dc959dc-ab62-40f7-81ea-e62e4aafed21 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:31.681970969 +0000 UTC m=+34.099789284 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret") pod "global-pull-secret-syncer-sj4n7" (UID: "9dc959dc-ab62-40f7-81ea-e62e4aafed21") : object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:15.782598 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:15.782567    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs\") pod \"network-metrics-daemon-rd7qq\" (UID: \"ca3463da-4922-4b81-a71e-8769f4a90494\") " pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:15.782761 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:15.782614    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9p6\" (UniqueName: \"kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6\") pod \"network-check-target-ngp84\" (UID: \"3d5cbc4c-5f2e-4611-85ee-70e51ebadd34\") " pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:15.782837 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:15.782755    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:15.782837 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:15.782768    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 23:37:15.782837 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:15.782790    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 23:37:15.782837 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:15.782803    2557 projected.go:196] Error preparing data for projected volume kube-api-access-gh9p6 for pod openshift-network-diagnostics/network-check-target-ngp84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:37:15.782837 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:15.782830    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs podName:ca3463da-4922-4b81-a71e-8769f4a90494 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:31.782809063 +0000 UTC m=+34.200627378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs") pod "network-metrics-daemon-rd7qq" (UID: "ca3463da-4922-4b81-a71e-8769f4a90494") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:15.783047 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:15.782853    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6 podName:3d5cbc4c-5f2e-4611-85ee-70e51ebadd34 nodeName:}" failed. No retries permitted until 2026-09-04 23:37:31.782836697 +0000 UTC m=+34.200654995 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gh9p6" (UniqueName: "kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6") pod "network-check-target-ngp84" (UID: "3d5cbc4c-5f2e-4611-85ee-70e51ebadd34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:37:16.073089 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:16.073005    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:16.073505 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:16.073165    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:16.073505 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:16.073211    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:16.073505 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:16.073322    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:17.072766 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:17.072730    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:17.072917 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:17.072869    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:18.073801 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:18.073756    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:18.074284 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:18.073881    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:18.074284 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:18.073903    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:18.074284 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:18.074035    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:19.073288 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.072999    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:19.073401 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:19.073370    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:19.170205 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.170177    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8mvzr" event={"ID":"fcc26e56-e58d-4030-9b3e-d87b573d6a8c","Type":"ContainerStarted","Data":"fb56061bed8ff839f9c852747fa611fdbe02450a4a4bf438775c79624757af8e"}
Sep 04 23:37:19.171568 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.171546    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rkpz9" event={"ID":"e356b35c-5280-4e59-9cd2-303af05624a3","Type":"ContainerStarted","Data":"a6329bf73cd14b2b78e350e91b821f0391a4b8d242bcdf51eaacd45b6191c7f8"}
Sep 04 23:37:19.173096 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.173075    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9" event={"ID":"0686ad4c-9eac-4756-a85d-8da561ee8e6a","Type":"ContainerStarted","Data":"82e775614fd1a7ec8ae004ffa6ee8091b1f7263b72c39f4683a769306be53517"}
Sep 04 23:37:19.174551 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.174527    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wkxtm" event={"ID":"d6a5b60d-4a14-49cc-8be4-f86989c0bd88","Type":"ContainerStarted","Data":"775b5b58ae70d8a2b7fef9a1f73b4c5007a88f04bae03b4053acee47855d97ee"}
Sep 04 23:37:19.175968 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.175931    2557 generic.go:356] "Generic (PLEG): container finished" podID="740e83e1-71de-4cca-ba79-614a1f20be97" containerID="14237846d349249c3c8685e0b353231115ba6a06735a0b719f6146e3852d7a29" exitCode=0
Sep 04 23:37:19.176043 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.175969    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg92b" event={"ID":"740e83e1-71de-4cca-ba79-614a1f20be97","Type":"ContainerDied","Data":"14237846d349249c3c8685e0b353231115ba6a06735a0b719f6146e3852d7a29"}
Sep 04 23:37:19.178312 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.178294    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 04 23:37:19.178629 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.178613    2557 generic.go:356] "Generic (PLEG): container finished" podID="5105066c-f4b4-450a-944c-e7dcd6964851" containerID="5fb4610b28c6a99a3a9c28c5e9fd216762ea6b6c938e44f60915c45a24417950" exitCode=1
Sep 04 23:37:19.178705 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.178685    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m" event={"ID":"5105066c-f4b4-450a-944c-e7dcd6964851","Type":"ContainerStarted","Data":"a743f5423ee4418169feffc4e55ca5b50aff144ca183e2c00ccbac773246b3d8"}
Sep 04 23:37:19.178780 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.178764    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m" event={"ID":"5105066c-f4b4-450a-944c-e7dcd6964851","Type":"ContainerStarted","Data":"05a6802adc1d0b225cb71b5abd4eccbb9be19e71639732583da5532f552a1d53"}
Sep 04 23:37:19.178832 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.178818    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m" event={"ID":"5105066c-f4b4-450a-944c-e7dcd6964851","Type":"ContainerDied","Data":"5fb4610b28c6a99a3a9c28c5e9fd216762ea6b6c938e44f60915c45a24417950"}
Sep 04 23:37:19.178886 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.178872    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m" event={"ID":"5105066c-f4b4-450a-944c-e7dcd6964851","Type":"ContainerStarted","Data":"43fb52f6c38039843eb8629b005a78af326d48b8c6e06ef34f4a8debf351f552"}
Sep 04 23:37:19.180136 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.180110    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-brgnz" event={"ID":"f60b1cfd-03dc-43f3-9cce-a0bbffbb4226","Type":"ContainerStarted","Data":"9638430bcc9e5364c40ad6ac3f74476ed827021e0c5ed1dbb931a26c87130b6a"}
Sep 04 23:37:19.181771 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.181752    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jjnzw" event={"ID":"cfd119fb-c2f0-458e-819e-f2de497ef5dd","Type":"ContainerStarted","Data":"7f8018955bdc756336dc609ca412695fd0e77cd8c0619d9e586a9fdcda778aeb"}
Sep 04 23:37:19.185585 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.185550    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" podStartSLOduration=20.185522673 podStartE2EDuration="20.185522673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:03.15247728 +0000 UTC m=+5.570295600" watchObservedRunningTime="2026-09-04 23:37:19.185522673 +0000 UTC m=+21.603340982"
Sep 04 23:37:19.186118 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.186094    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8mvzr" podStartSLOduration=3.546702742 podStartE2EDuration="21.186084757s" totalImagesPullingTime="17.639382015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:19.18565328 +0000 UTC m=+21.603471575" watchObservedRunningTime="2026-09-04 23:37:19.186084757 +0000 UTC m=+21.603903074"
Sep 04 23:37:19.201253 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.201206    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rkpz9" podStartSLOduration=10.981957572 podStartE2EDuration="21.201187373s" totalImagesPullingTime="10.219229801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:19.201089084 +0000 UTC m=+21.618907402" watchObservedRunningTime="2026-09-04 23:37:19.201187373 +0000 UTC m=+21.619005694"
Sep 04 23:37:19.215577 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.215542    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wkxtm" podStartSLOduration=3.585356375 podStartE2EDuration="21.215529013s" totalImagesPullingTime="17.630172638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:19.215184883 +0000 UTC m=+21.633003200" watchObservedRunningTime="2026-09-04 23:37:19.215529013 +0000 UTC m=+21.633347361"
Sep 04 23:37:19.234798 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.234760    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jjnzw" podStartSLOduration=3.39565013 podStartE2EDuration="21.23474955s" totalImagesPullingTime="17.83909942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:19.234588906 +0000 UTC m=+21.652407223" watchObservedRunningTime="2026-09-04 23:37:19.23474955 +0000 UTC m=+21.652567867"
Sep 04 23:37:19.248536 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:19.248503    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-brgnz" podStartSLOduration=3.611469916 podStartE2EDuration="21.248491129s" totalImagesPullingTime="17.637021213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:19.248091625 +0000 UTC m=+21.665909942" watchObservedRunningTime="2026-09-04 23:37:19.248491129 +0000 UTC m=+21.666309447"
Sep 04 23:37:20.073380 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:20.073354    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:20.073492 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:20.073395    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:20.073492 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:20.073481    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:20.073605 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:20.073561    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:20.092721 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:20.092698    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rkpz9"
Sep 04 23:37:20.189573 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:20.189554    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 04 23:37:20.190077 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:20.190055    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m" event={"ID":"5105066c-f4b4-450a-944c-e7dcd6964851","Type":"ContainerStarted","Data":"eed98cc3e6efd8ce6bc755b1267f5f51ec31a4347840b150787ff09841edb4ec"}
Sep 04 23:37:20.190176 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:20.190135    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m" event={"ID":"5105066c-f4b4-450a-944c-e7dcd6964851","Type":"ContainerStarted","Data":"c2855aa366da3df0fc33af247c71798203bda8fb97798f0ad3c96bd3c3982f7e"}
Sep 04 23:37:20.192160 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:20.192136    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-42mcf" event={"ID":"ccabb3bd-e9c8-4b72-8bcc-9c7a15cb6790","Type":"ContainerStarted","Data":"f6585d5145da6b97e6f4a5b6f43381348ee211c5c353b6e0cefeb7215fb958cd"}
Sep 04 23:37:20.196162 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:20.196142    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 04 23:37:20.206213 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:20.206177    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-42mcf" podStartSLOduration=4.561376956 podStartE2EDuration="22.2061625s" totalImagesPullingTime="17.644785544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:20.205785668 +0000 UTC m=+22.623603987" watchObservedRunningTime="2026-09-04 23:37:20.2061625 +0000 UTC m=+22.623980829"
Sep 04 23:37:20.999231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:20.999078    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-04T23:37:20.196158922Z","UUID":"c26962f4-08b8-4b1e-b546-2b91453b4f44","Handler":null,"Name":"","Endpoint":""}
Sep 04 23:37:21.000895 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:21.000863    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 04 23:37:21.001045 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:21.000917    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 04 23:37:21.073484 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:21.073459    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:21.073684 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:21.073632    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:21.196081 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:21.196047    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9" event={"ID":"0686ad4c-9eac-4756-a85d-8da561ee8e6a","Type":"ContainerStarted","Data":"a414d71fa09a2933428f502c18b60cd5e3686fc7ab71cbc92cd996145512f4b5"}
Sep 04 23:37:21.490965 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:21.490879    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rkpz9"
Sep 04 23:37:21.491545 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:21.491486    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rkpz9"
Sep 04 23:37:22.073393 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:22.073199    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:22.073563 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:22.073199    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:22.073563 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:22.073470    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:22.073685 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:22.073569    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:22.201729 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:22.201705    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 04 23:37:22.202186 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:22.202164    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m" event={"ID":"5105066c-f4b4-450a-944c-e7dcd6964851","Type":"ContainerStarted","Data":"8796dad32aa11fc458883e64821fce3b937fd41bba4df165bf713d7410c88e3b"}
Sep 04 23:37:22.204643 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:22.204617    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9" event={"ID":"0686ad4c-9eac-4756-a85d-8da561ee8e6a","Type":"ContainerStarted","Data":"8d172860a2aae0beb40b3c97e6b03b26b2f6246ff68f127e3a19b68d651bc459"}
Sep 04 23:37:22.205354 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:22.205330    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rkpz9"
Sep 04 23:37:22.222219 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:22.222177    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpvr9" podStartSLOduration=3.99969706 podStartE2EDuration="24.222164413s" totalImagesPullingTime="20.222467353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:22.221500256 +0000 UTC m=+24.639318585" watchObservedRunningTime="2026-09-04 23:37:22.222164413 +0000 UTC m=+24.639982730"
Sep 04 23:37:23.072809 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:23.072778    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:23.073012 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:23.072896    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:24.073312 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:24.073086    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:24.073870 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:24.073090    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:24.073870 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:24.073327    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:24.073870 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:24.073425    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:24.211481 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:24.211448    2557 generic.go:356] "Generic (PLEG): container finished" podID="740e83e1-71de-4cca-ba79-614a1f20be97" containerID="e32669dd4c7deac9eebdd991cf4e187d598fbdae547730f512bbb5f08f67ab3d" exitCode=0
Sep 04 23:37:24.211635 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:24.211537    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg92b" event={"ID":"740e83e1-71de-4cca-ba79-614a1f20be97","Type":"ContainerDied","Data":"e32669dd4c7deac9eebdd991cf4e187d598fbdae547730f512bbb5f08f67ab3d"}
Sep 04 23:37:24.215314 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:24.215296    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 04 23:37:24.215659 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:24.215641    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m" event={"ID":"5105066c-f4b4-450a-944c-e7dcd6964851","Type":"ContainerStarted","Data":"fd82bbe2de7611b7ccc080586df536dfd3c56018c2e00d5fc6e53e463a4787db"}
Sep 04 23:37:24.215982 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:24.215960    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:24.218770 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:24.216815    2557 scope.go:117] "RemoveContainer" containerID="5fb4610b28c6a99a3a9c28c5e9fd216762ea6b6c938e44f60915c45a24417950"
Sep 04 23:37:24.235488 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:24.235465    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:25.072555 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:25.072522    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:25.072696 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:25.072630    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:25.221713 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:25.221689    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 04 23:37:25.222078 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:25.222033    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m" event={"ID":"5105066c-f4b4-450a-944c-e7dcd6964851","Type":"ContainerStarted","Data":"312b2de4ab076173c6820a5f5f0d6006a406d3b627603b9e67177bec9c19b907"}
Sep 04 23:37:25.222315 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:25.222278    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:25.222416 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:25.222352    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:25.236696 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:25.236677    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:25.248807 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:25.248771    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m" podStartSLOduration=9.57904743 podStartE2EDuration="27.248760203s" totalImagesPullingTime="17.669712773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:25.246668691 +0000 UTC m=+27.664487009" watchObservedRunningTime="2026-09-04 23:37:25.248760203 +0000 UTC m=+27.666578521"
Sep 04 23:37:26.075725 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:26.075694    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:26.075896 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:26.075777    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:26.075896 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:26.075869    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:26.076072 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:26.075997    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:26.225748 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:26.225722    2557 generic.go:356] "Generic (PLEG): container finished" podID="740e83e1-71de-4cca-ba79-614a1f20be97" containerID="95d0e2a6d9781d7361a7fc2720b9e4a587598ff8198816f9901e2b81448cbc69" exitCode=0
Sep 04 23:37:26.226094 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:26.225788    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg92b" event={"ID":"740e83e1-71de-4cca-ba79-614a1f20be97","Type":"ContainerDied","Data":"95d0e2a6d9781d7361a7fc2720b9e4a587598ff8198816f9901e2b81448cbc69"}
Sep 04 23:37:27.072811 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:27.072793    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:27.072968 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:27.072902    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:27.230607 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:27.230582    2557 generic.go:356] "Generic (PLEG): container finished" podID="740e83e1-71de-4cca-ba79-614a1f20be97" containerID="e140797f3becc2ffb7e3cbc3cf7b48ad52cfa7d6dfecbdb15a7a351431de02d8" exitCode=0
Sep 04 23:37:27.230932 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:27.230658    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg92b" event={"ID":"740e83e1-71de-4cca-ba79-614a1f20be97","Type":"ContainerDied","Data":"e140797f3becc2ffb7e3cbc3cf7b48ad52cfa7d6dfecbdb15a7a351431de02d8"}
Sep 04 23:37:28.073863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:28.073826    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:28.074021 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:28.073924    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:28.074021 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:28.073995    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:28.074141 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:28.074049    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:29.073155 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:29.073117    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:29.073643 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:29.073273    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:30.076458 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:30.076424    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:30.076828 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:30.076424    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:30.076828 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:30.076530    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:30.076828 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:30.076597    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:31.072886 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:31.072853    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:31.073182 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:31.072994    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:31.709746 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:31.709723    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:31.709994 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:31.709815    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:31.709994 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:31.709860    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret podName:9dc959dc-ab62-40f7-81ea-e62e4aafed21 nodeName:}" failed. No retries permitted until 2026-09-04 23:38:03.709847455 +0000 UTC m=+66.127665765 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret") pod "global-pull-secret-syncer-sj4n7" (UID: "9dc959dc-ab62-40f7-81ea-e62e4aafed21") : object "kube-system"/"original-pull-secret" not registered
Sep 04 23:37:31.810846 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:31.810806    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs\") pod \"network-metrics-daemon-rd7qq\" (UID: \"ca3463da-4922-4b81-a71e-8769f4a90494\") " pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:31.810846 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:31.810847    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9p6\" (UniqueName: \"kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6\") pod \"network-check-target-ngp84\" (UID: \"3d5cbc4c-5f2e-4611-85ee-70e51ebadd34\") " pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:31.811103 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:31.810964    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:31.811103 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:31.810975    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 23:37:31.811103 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:31.810993    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 23:37:31.811103 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:31.811002    2557 projected.go:196] Error preparing data for projected volume kube-api-access-gh9p6 for pod openshift-network-diagnostics/network-check-target-ngp84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:37:31.811103 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:31.811028    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs podName:ca3463da-4922-4b81-a71e-8769f4a90494 nodeName:}" failed. No retries permitted until 2026-09-04 23:38:03.811012394 +0000 UTC m=+66.228830710 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs") pod "network-metrics-daemon-rd7qq" (UID: "ca3463da-4922-4b81-a71e-8769f4a90494") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 23:37:31.811103 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:31.811043    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6 podName:3d5cbc4c-5f2e-4611-85ee-70e51ebadd34 nodeName:}" failed. No retries permitted until 2026-09-04 23:38:03.811036883 +0000 UTC m=+66.228855179 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gh9p6" (UniqueName: "kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6") pod "network-check-target-ngp84" (UID: "3d5cbc4c-5f2e-4611-85ee-70e51ebadd34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 23:37:32.076992 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:32.076962    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:32.077141 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:32.076995    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:32.077141 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:32.077087    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:32.077232 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:32.077216    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:32.244850 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:32.244820    2557 generic.go:356] "Generic (PLEG): container finished" podID="740e83e1-71de-4cca-ba79-614a1f20be97" containerID="2651085d1af26b2e871c90bc85bd21d424ec1f6bf9498c10624b0a5f5dbdd3ed" exitCode=0
Sep 04 23:37:32.245023 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:32.244863    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg92b" event={"ID":"740e83e1-71de-4cca-ba79-614a1f20be97","Type":"ContainerDied","Data":"2651085d1af26b2e871c90bc85bd21d424ec1f6bf9498c10624b0a5f5dbdd3ed"}
Sep 04 23:37:32.812753 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:32.812561    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sj4n7"]
Sep 04 23:37:32.813426 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:32.812885    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:32.813426 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:32.813011    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:32.815682 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:32.815656    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ngp84"]
Sep 04 23:37:32.816552 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:32.815771    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:32.816552 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:32.815888    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:32.816552 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:32.816264    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rd7qq"]
Sep 04 23:37:32.816552 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:32.816360    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:32.816552 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:32.816467    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:33.249981 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:33.249898    2557 generic.go:356] "Generic (PLEG): container finished" podID="740e83e1-71de-4cca-ba79-614a1f20be97" containerID="041f365d272bb4540acfc28f8c79dc8cf0ab9f728ffd232fa5143d921a0b68cd" exitCode=0
Sep 04 23:37:33.250125 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:33.249980    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg92b" event={"ID":"740e83e1-71de-4cca-ba79-614a1f20be97","Type":"ContainerDied","Data":"041f365d272bb4540acfc28f8c79dc8cf0ab9f728ffd232fa5143d921a0b68cd"}
Sep 04 23:37:34.075734 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:34.075708    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:34.076341 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:34.075827    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:34.076341 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:34.075874    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:34.076341 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:34.075998    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:34.254924 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:34.254898    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg92b" event={"ID":"740e83e1-71de-4cca-ba79-614a1f20be97","Type":"ContainerStarted","Data":"6a6726cf20c122cea29ce6bbe56b925ec8417f66a7abfbb9d593c442078a7526"}
Sep 04 23:37:34.276260 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:34.276217    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wg92b" podStartSLOduration=7.910590779 podStartE2EDuration="36.276203966s" totalImagesPullingTime="28.365613187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:34.27581248 +0000 UTC m=+36.693630797" watchObservedRunningTime="2026-09-04 23:37:34.276203966 +0000 UTC m=+36.694022283"
Sep 04 23:37:35.072863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:35.072836    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:35.073029 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:35.072919    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:36.072859 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:36.072827    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:36.073318 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:36.072932    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sj4n7" podUID="9dc959dc-ab62-40f7-81ea-e62e4aafed21"
Sep 04 23:37:36.073318 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:36.072989    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:36.073318 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:36.073086    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd7qq" podUID="ca3463da-4922-4b81-a71e-8769f4a90494"
Sep 04 23:37:37.073457 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.073384    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:37.073789 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:37.073474    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngp84" podUID="3d5cbc4c-5f2e-4611-85ee-70e51ebadd34"
Sep 04 23:37:37.386856 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.386801    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeReady"
Sep 04 23:37:37.386967 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.386888    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 04 23:37:37.419072 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.419045    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-766d6d64dc-7h6zs"]
Sep 04 23:37:37.442966 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.442925    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-crz7n"]
Sep 04 23:37:37.443085 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.443069    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.446557 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.446066    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 04 23:37:37.446557 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.446230    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 04 23:37:37.446557 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.446383    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6hx2z\""
Sep 04 23:37:37.446557 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.446499    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 04 23:37:37.452913 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.452891    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 04 23:37:37.459036 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.459018    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-766d6d64dc-7h6zs"]
Sep 04 23:37:37.459116 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.459047    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kh7ws"]
Sep 04 23:37:37.459157 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.459115    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:37.461947 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.461917    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 04 23:37:37.462047 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.461918    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4rhln\""
Sep 04 23:37:37.462109 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.462094    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 04 23:37:37.477272 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.477254    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-crz7n"]
Sep 04 23:37:37.477354 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.477283    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kh7ws"]
Sep 04 23:37:37.477354 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.477348    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kh7ws"
Sep 04 23:37:37.479843 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.479824    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 04 23:37:37.479962 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.479869    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 04 23:37:37.479962 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.479912    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ttwxb\""
Sep 04 23:37:37.480069 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.479969    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 04 23:37:37.554540 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.554521    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69786b86-8ed4-44a2-805a-3021d4a5dc6f-image-registry-private-configuration\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.554621 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.554545    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69786b86-8ed4-44a2-805a-3021d4a5dc6f-installation-pull-secrets\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.554621 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.554576    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69786b86-8ed4-44a2-805a-3021d4a5dc6f-ca-trust-extracted\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.554621 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.554590    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df6dg\" (UniqueName: \"kubernetes.io/projected/69786b86-8ed4-44a2-805a-3021d4a5dc6f-kube-api-access-df6dg\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.554729 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.554668    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69786b86-8ed4-44a2-805a-3021d4a5dc6f-trusted-ca\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.554729 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.554700    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69786b86-8ed4-44a2-805a-3021d4a5dc6f-registry-tls\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.554729 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.554724    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69786b86-8ed4-44a2-805a-3021d4a5dc6f-registry-certificates\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.554816 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.554742    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69786b86-8ed4-44a2-805a-3021d4a5dc6f-bound-sa-token\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.655890 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.655845    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94wgf\" (UniqueName: \"kubernetes.io/projected/9f4ed71e-4b2e-4672-b7b8-832b3f4993de-kube-api-access-94wgf\") pod \"ingress-canary-kh7ws\" (UID: \"9f4ed71e-4b2e-4672-b7b8-832b3f4993de\") " pod="openshift-ingress-canary/ingress-canary-kh7ws"
Sep 04 23:37:37.655890 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.655876    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69786b86-8ed4-44a2-805a-3021d4a5dc6f-trusted-ca\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.655890 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.655893    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69786b86-8ed4-44a2-805a-3021d4a5dc6f-registry-tls\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.656055 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.655958    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69786b86-8ed4-44a2-805a-3021d4a5dc6f-registry-certificates\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.656055 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.656002    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69786b86-8ed4-44a2-805a-3021d4a5dc6f-bound-sa-token\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.656055 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.656037    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69786b86-8ed4-44a2-805a-3021d4a5dc6f-image-registry-private-configuration\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.656055 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.656053    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69786b86-8ed4-44a2-805a-3021d4a5dc6f-installation-pull-secrets\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.656237 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.656093    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d83926d9-914f-4bff-99ba-6ba74b32ada5-config-volume\") pod \"dns-default-crz7n\" (UID: \"d83926d9-914f-4bff-99ba-6ba74b32ada5\") " pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:37.656237 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.656117    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83926d9-914f-4bff-99ba-6ba74b32ada5-tmp-dir\") pod \"dns-default-crz7n\" (UID: \"d83926d9-914f-4bff-99ba-6ba74b32ada5\") " pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:37.656237 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.656131    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxzhf\" (UniqueName: \"kubernetes.io/projected/d83926d9-914f-4bff-99ba-6ba74b32ada5-kube-api-access-vxzhf\") pod \"dns-default-crz7n\" (UID: \"d83926d9-914f-4bff-99ba-6ba74b32ada5\") " pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:37.656237 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.656151    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69786b86-8ed4-44a2-805a-3021d4a5dc6f-ca-trust-extracted\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.656237 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.656173    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6dg\" (UniqueName: \"kubernetes.io/projected/69786b86-8ed4-44a2-805a-3021d4a5dc6f-kube-api-access-df6dg\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.656237 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.656197    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d83926d9-914f-4bff-99ba-6ba74b32ada5-metrics-tls\") pod \"dns-default-crz7n\" (UID: \"d83926d9-914f-4bff-99ba-6ba74b32ada5\") " pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:37.656237 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.656238    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f4ed71e-4b2e-4672-b7b8-832b3f4993de-cert\") pod \"ingress-canary-kh7ws\" (UID: \"9f4ed71e-4b2e-4672-b7b8-832b3f4993de\") " pod="openshift-ingress-canary/ingress-canary-kh7ws"
Sep 04 23:37:37.656619 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.656597    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69786b86-8ed4-44a2-805a-3021d4a5dc6f-ca-trust-extracted\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.656908 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.656888    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69786b86-8ed4-44a2-805a-3021d4a5dc6f-registry-certificates\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.656991 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.656906    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69786b86-8ed4-44a2-805a-3021d4a5dc6f-trusted-ca\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.659743 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.659726    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69786b86-8ed4-44a2-805a-3021d4a5dc6f-image-registry-private-configuration\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.659743 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.659742    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69786b86-8ed4-44a2-805a-3021d4a5dc6f-installation-pull-secrets\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.659928 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.659910    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69786b86-8ed4-44a2-805a-3021d4a5dc6f-registry-tls\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.663791 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.663766    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df6dg\" (UniqueName: \"kubernetes.io/projected/69786b86-8ed4-44a2-805a-3021d4a5dc6f-kube-api-access-df6dg\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.663884 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.663788    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69786b86-8ed4-44a2-805a-3021d4a5dc6f-bound-sa-token\") pod \"image-registry-766d6d64dc-7h6zs\" (UID: \"69786b86-8ed4-44a2-805a-3021d4a5dc6f\") " pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.754653 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.754609    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:37.756840 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.756822    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f4ed71e-4b2e-4672-b7b8-832b3f4993de-cert\") pod \"ingress-canary-kh7ws\" (UID: \"9f4ed71e-4b2e-4672-b7b8-832b3f4993de\") " pod="openshift-ingress-canary/ingress-canary-kh7ws"
Sep 04 23:37:37.756900 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.756852    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94wgf\" (UniqueName: \"kubernetes.io/projected/9f4ed71e-4b2e-4672-b7b8-832b3f4993de-kube-api-access-94wgf\") pod \"ingress-canary-kh7ws\" (UID: \"9f4ed71e-4b2e-4672-b7b8-832b3f4993de\") " pod="openshift-ingress-canary/ingress-canary-kh7ws"
Sep 04 23:37:37.757003 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.756985    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d83926d9-914f-4bff-99ba-6ba74b32ada5-config-volume\") pod \"dns-default-crz7n\" (UID: \"d83926d9-914f-4bff-99ba-6ba74b32ada5\") " pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:37.757052 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.757011    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83926d9-914f-4bff-99ba-6ba74b32ada5-tmp-dir\") pod \"dns-default-crz7n\" (UID: \"d83926d9-914f-4bff-99ba-6ba74b32ada5\") " pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:37.757052 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.757029    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxzhf\" (UniqueName: \"kubernetes.io/projected/d83926d9-914f-4bff-99ba-6ba74b32ada5-kube-api-access-vxzhf\") pod \"dns-default-crz7n\" (UID: \"d83926d9-914f-4bff-99ba-6ba74b32ada5\") " pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:37.757052 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.757048    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d83926d9-914f-4bff-99ba-6ba74b32ada5-metrics-tls\") pod \"dns-default-crz7n\" (UID: \"d83926d9-914f-4bff-99ba-6ba74b32ada5\") " pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:37.757342 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.757320    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83926d9-914f-4bff-99ba-6ba74b32ada5-tmp-dir\") pod \"dns-default-crz7n\" (UID: \"d83926d9-914f-4bff-99ba-6ba74b32ada5\") " pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:37.757602 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.757583    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d83926d9-914f-4bff-99ba-6ba74b32ada5-config-volume\") pod \"dns-default-crz7n\" (UID: \"d83926d9-914f-4bff-99ba-6ba74b32ada5\") " pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:37.758887 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.758864    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f4ed71e-4b2e-4672-b7b8-832b3f4993de-cert\") pod \"ingress-canary-kh7ws\" (UID: \"9f4ed71e-4b2e-4672-b7b8-832b3f4993de\") " pod="openshift-ingress-canary/ingress-canary-kh7ws"
Sep 04 23:37:37.759116 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.759098    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d83926d9-914f-4bff-99ba-6ba74b32ada5-metrics-tls\") pod \"dns-default-crz7n\" (UID: \"d83926d9-914f-4bff-99ba-6ba74b32ada5\") " pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:37.764316 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.764291    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94wgf\" (UniqueName: \"kubernetes.io/projected/9f4ed71e-4b2e-4672-b7b8-832b3f4993de-kube-api-access-94wgf\") pod \"ingress-canary-kh7ws\" (UID: \"9f4ed71e-4b2e-4672-b7b8-832b3f4993de\") " pod="openshift-ingress-canary/ingress-canary-kh7ws"
Sep 04 23:37:37.764420 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.764382    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxzhf\" (UniqueName: \"kubernetes.io/projected/d83926d9-914f-4bff-99ba-6ba74b32ada5-kube-api-access-vxzhf\") pod \"dns-default-crz7n\" (UID: \"d83926d9-914f-4bff-99ba-6ba74b32ada5\") " pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:37.767169 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.767153    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:37.785167 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:37.785145    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kh7ws"
Sep 04 23:37:38.070777 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:38.070756    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-766d6d64dc-7h6zs"]
Sep 04 23:37:38.072921 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:37:38.072898    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69786b86_8ed4_44a2_805a_3021d4a5dc6f.slice/crio-e650a28b6de3dbc5083316796f53410bc137d40c6b99e5016676602274cc7061 WatchSource:0}: Error finding container e650a28b6de3dbc5083316796f53410bc137d40c6b99e5016676602274cc7061: Status 404 returned error can't find the container with id e650a28b6de3dbc5083316796f53410bc137d40c6b99e5016676602274cc7061
Sep 04 23:37:38.074465 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:38.074439    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:37:38.079195 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:38.074854    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:37:38.079195 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:38.077395    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8jjtb\""
Sep 04 23:37:38.079195 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:38.078408    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 04 23:37:38.079195 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:37:38.078660    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83926d9_914f_4bff_99ba_6ba74b32ada5.slice/crio-91448c82e4870413979348cd07a8218a8b019b0551af8de4cf8d442261b1df1b WatchSource:0}: Error finding container 91448c82e4870413979348cd07a8218a8b019b0551af8de4cf8d442261b1df1b: Status 404 returned error can't find the container with id 91448c82e4870413979348cd07a8218a8b019b0551af8de4cf8d442261b1df1b
Sep 04 23:37:38.079679 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:38.079403    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 04 23:37:38.080216 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:38.080195    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-crz7n"]
Sep 04 23:37:38.101041 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:38.101022    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kh7ws"]
Sep 04 23:37:38.103401 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:37:38.103381    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4ed71e_4b2e_4672_b7b8_832b3f4993de.slice/crio-4e94e484d4c9dfad1ee9c27a4e3f37a145d617e3d571e328db1ccc51b1dd0d7c WatchSource:0}: Error finding container 4e94e484d4c9dfad1ee9c27a4e3f37a145d617e3d571e328db1ccc51b1dd0d7c: Status 404 returned error can't find the container with id 4e94e484d4c9dfad1ee9c27a4e3f37a145d617e3d571e328db1ccc51b1dd0d7c
Sep 04 23:37:38.264218 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:38.264190    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kh7ws" event={"ID":"9f4ed71e-4b2e-4672-b7b8-832b3f4993de","Type":"ContainerStarted","Data":"4e94e484d4c9dfad1ee9c27a4e3f37a145d617e3d571e328db1ccc51b1dd0d7c"}
Sep 04 23:37:38.265296 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:38.265276    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-crz7n" event={"ID":"d83926d9-914f-4bff-99ba-6ba74b32ada5","Type":"ContainerStarted","Data":"91448c82e4870413979348cd07a8218a8b019b0551af8de4cf8d442261b1df1b"}
Sep 04 23:37:38.266587 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:38.266566    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs" event={"ID":"69786b86-8ed4-44a2-805a-3021d4a5dc6f","Type":"ContainerStarted","Data":"4b02cad80bc28df3a737796fa5026d0b5c42d481c5b4b4720a8a34f5008eea11"}
Sep 04 23:37:38.266664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:38.266606    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs" event={"ID":"69786b86-8ed4-44a2-805a-3021d4a5dc6f","Type":"ContainerStarted","Data":"e650a28b6de3dbc5083316796f53410bc137d40c6b99e5016676602274cc7061"}
Sep 04 23:37:38.266789 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:38.266771    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:38.284308 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:38.284263    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs" podStartSLOduration=6.284246904 podStartE2EDuration="6.284246904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:38.283609279 +0000 UTC m=+40.701427605" watchObservedRunningTime="2026-09-04 23:37:38.284246904 +0000 UTC m=+40.702065223"
Sep 04 23:37:39.073174 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:39.073139    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:37:39.075915 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:39.075891    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 04 23:37:39.076311 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:39.075927    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vhhzf\""
Sep 04 23:37:39.076311 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:39.075891    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 04 23:37:42.278316 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:42.278283    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kh7ws" event={"ID":"9f4ed71e-4b2e-4672-b7b8-832b3f4993de","Type":"ContainerStarted","Data":"8f10b6446530f9cc123910efafae431e195e735118ffb0a648b2f9c3b6e177d0"}
Sep 04 23:37:42.280072 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:42.280046    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-crz7n" event={"ID":"d83926d9-914f-4bff-99ba-6ba74b32ada5","Type":"ContainerStarted","Data":"14cf7ab1dca4a1d0c3ec5ea82502f171de54ac0287e640c76975d7c5991c9fdc"}
Sep 04 23:37:42.280188 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:42.280101    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:42.280188 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:42.280120    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-crz7n" event={"ID":"d83926d9-914f-4bff-99ba-6ba74b32ada5","Type":"ContainerStarted","Data":"e2b3209d4199931edcd4474719c94275f7c750d68a8b9593fa77ae567d911686"}
Sep 04 23:37:42.296028 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:42.295988    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kh7ws" podStartSLOduration=2.149390844 podStartE2EDuration="5.295975322s" totalImagesPullingTime="3.146584478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:42.295772483 +0000 UTC m=+44.713590800" watchObservedRunningTime="2026-09-04 23:37:42.295975322 +0000 UTC m=+44.713793634"
Sep 04 23:37:42.318663 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:42.318622    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-crz7n" podStartSLOduration=2.152717947 podStartE2EDuration="5.318612775s" totalImagesPullingTime="3.165894828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:37:42.318030553 +0000 UTC m=+44.735848872" watchObservedRunningTime="2026-09-04 23:37:42.318612775 +0000 UTC m=+44.736431093"
Sep 04 23:37:52.284597 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:52.284563    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-crz7n"
Sep 04 23:37:54.859605 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:54.859573    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l8th6"]
Sep 04 23:37:54.892359 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:54.892327    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l8th6"]
Sep 04 23:37:54.892487 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:54.892442    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:54.895000 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:54.894963    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 04 23:37:54.895132 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:54.895027    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 04 23:37:54.895132 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:54.895043    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 04 23:37:54.895132 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:54.895047    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 04 23:37:54.895287 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:54.895217    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qcf5d\""
Sep 04 23:37:54.969027 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:54.968995    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l92mw\" (UniqueName: \"kubernetes.io/projected/514da723-8019-4825-a454-a1d56c30be6f-kube-api-access-l92mw\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:54.969139 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:54.969048    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/514da723-8019-4825-a454-a1d56c30be6f-data-volume\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:54.969186 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:54.969135    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:54.969186 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:54.969161    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:54.969257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:54.969187    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/514da723-8019-4825-a454-a1d56c30be6f-crio-socket\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:55.069957 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:55.069908    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:55.070092 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:55.069970    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:55.070092 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:55.070004    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/514da723-8019-4825-a454-a1d56c30be6f-crio-socket\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:55.070092 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:55.070031    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 23:37:55.070261 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:55.070040    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l92mw\" (UniqueName: \"kubernetes.io/projected/514da723-8019-4825-a454-a1d56c30be6f-kube-api-access-l92mw\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:55.070261 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:55.070101    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm podName:514da723-8019-4825-a454-a1d56c30be6f nodeName:}" failed. No retries permitted until 2026-09-04 23:37:55.570081404 +0000 UTC m=+57.987899702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8th6" (UID: "514da723-8019-4825-a454-a1d56c30be6f") : configmap "kube-rbac-proxy" not found
Sep 04 23:37:55.070261 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:55.070130    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:55.070261 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:55.070141    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/514da723-8019-4825-a454-a1d56c30be6f-data-volume\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:55.070261 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:55.070130    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/514da723-8019-4825-a454-a1d56c30be6f-crio-socket\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:55.070261 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:55.070180    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls podName:514da723-8019-4825-a454-a1d56c30be6f nodeName:}" failed. No retries permitted until 2026-09-04 23:37:55.570165105 +0000 UTC m=+57.987983411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8th6" (UID: "514da723-8019-4825-a454-a1d56c30be6f") : secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:55.070472 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:55.070456    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/514da723-8019-4825-a454-a1d56c30be6f-data-volume\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:55.082101 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:55.082079    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l92mw\" (UniqueName: \"kubernetes.io/projected/514da723-8019-4825-a454-a1d56c30be6f-kube-api-access-l92mw\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:55.573371 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:55.573339    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:55.573371 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:55.573375    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:55.573597 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:55.573463    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 23:37:55.573597 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:55.573528    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm podName:514da723-8019-4825-a454-a1d56c30be6f nodeName:}" failed. No retries permitted until 2026-09-04 23:37:56.573511355 +0000 UTC m=+58.991329651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8th6" (UID: "514da723-8019-4825-a454-a1d56c30be6f") : configmap "kube-rbac-proxy" not found
Sep 04 23:37:55.573597 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:55.573472    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:55.573726 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:55.573623    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls podName:514da723-8019-4825-a454-a1d56c30be6f nodeName:}" failed. No retries permitted until 2026-09-04 23:37:56.573604072 +0000 UTC m=+58.991422372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8th6" (UID: "514da723-8019-4825-a454-a1d56c30be6f") : secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:56.580480 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:56.580440    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:56.580480 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:56.580481    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:56.580871 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:56.580557    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 23:37:56.580871 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:56.580590    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:56.580871 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:56.580619    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm podName:514da723-8019-4825-a454-a1d56c30be6f nodeName:}" failed. No retries permitted until 2026-09-04 23:37:58.580602654 +0000 UTC m=+60.998420955 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8th6" (UID: "514da723-8019-4825-a454-a1d56c30be6f") : configmap "kube-rbac-proxy" not found
Sep 04 23:37:56.580871 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:56.580633    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls podName:514da723-8019-4825-a454-a1d56c30be6f nodeName:}" failed. No retries permitted until 2026-09-04 23:37:58.580626784 +0000 UTC m=+60.998445079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8th6" (UID: "514da723-8019-4825-a454-a1d56c30be6f") : secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:57.240760 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:57.240736    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6m5m"
Sep 04 23:37:57.758797 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:57.758760    2557 patch_prober.go:28] interesting pod/image-registry-766d6d64dc-7h6zs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 04 23:37:57.759158 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:57.758811    2557 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs" podUID="69786b86-8ed4-44a2-805a-3021d4a5dc6f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 23:37:57.852438 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:57.852414    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-crz7n_d83926d9-914f-4bff-99ba-6ba74b32ada5/dns/0.log"
Sep 04 23:37:58.040128 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:58.040063    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-crz7n_d83926d9-914f-4bff-99ba-6ba74b32ada5/kube-rbac-proxy/0.log"
Sep 04 23:37:58.595195 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:58.595158    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:58.595368 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:58.595269    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:37:58.595368 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:58.595297    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:58.595368 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:58.595351    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 23:37:58.595368 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:58.595366    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls podName:514da723-8019-4825-a454-a1d56c30be6f nodeName:}" failed. No retries permitted until 2026-09-04 23:38:02.595350378 +0000 UTC m=+65.013168678 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8th6" (UID: "514da723-8019-4825-a454-a1d56c30be6f") : secret "insights-runtime-extractor-tls" not found
Sep 04 23:37:58.595534 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:37:58.595401    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm podName:514da723-8019-4825-a454-a1d56c30be6f nodeName:}" failed. No retries permitted until 2026-09-04 23:38:02.595384524 +0000 UTC m=+65.013202837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8th6" (UID: "514da723-8019-4825-a454-a1d56c30be6f") : configmap "kube-rbac-proxy" not found
Sep 04 23:37:59.237249 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:59.237215    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-brgnz_f60b1cfd-03dc-43f3-9cce-a0bbffbb4226/dns-node-resolver/0.log"
Sep 04 23:37:59.274090 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:59.274065    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-766d6d64dc-7h6zs"
Sep 04 23:37:59.637268 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:37:59.637202    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-766d6d64dc-7h6zs_69786b86-8ed4-44a2-805a-3021d4a5dc6f/registry/0.log"
Sep 04 23:38:00.436847 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:00.436823    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wkxtm_d6a5b60d-4a14-49cc-8be4-f86989c0bd88/node-ca/0.log"
Sep 04 23:38:01.037080 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:01.037056    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kh7ws_9f4ed71e-4b2e-4672-b7b8-832b3f4993de/serve-healthcheck-canary/0.log"
Sep 04 23:38:02.625210 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:02.625177    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:38:02.625210 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:02.625210    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:38:02.625602 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:38:02.625294    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 23:38:02.625602 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:38:02.625313    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 23:38:02.625602 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:38:02.625353    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls podName:514da723-8019-4825-a454-a1d56c30be6f nodeName:}" failed. No retries permitted until 2026-09-04 23:38:10.625340727 +0000 UTC m=+73.043159023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8th6" (UID: "514da723-8019-4825-a454-a1d56c30be6f") : secret "insights-runtime-extractor-tls" not found
Sep 04 23:38:02.625602 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:38:02.625365    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm podName:514da723-8019-4825-a454-a1d56c30be6f nodeName:}" failed. No retries permitted until 2026-09-04 23:38:10.625359337 +0000 UTC m=+73.043177632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8th6" (UID: "514da723-8019-4825-a454-a1d56c30be6f") : configmap "kube-rbac-proxy" not found
Sep 04 23:38:03.734455 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:03.734420    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:38:03.737104 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:03.737087    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 04 23:38:03.747223 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:03.747202    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9dc959dc-ab62-40f7-81ea-e62e4aafed21-original-pull-secret\") pod \"global-pull-secret-syncer-sj4n7\" (UID: \"9dc959dc-ab62-40f7-81ea-e62e4aafed21\") " pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:38:03.835530 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:03.835501    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs\") pod \"network-metrics-daemon-rd7qq\" (UID: \"ca3463da-4922-4b81-a71e-8769f4a90494\") " pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:38:03.835642 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:03.835553    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9p6\" (UniqueName: \"kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6\") pod \"network-check-target-ngp84\" (UID: \"3d5cbc4c-5f2e-4611-85ee-70e51ebadd34\") " pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:38:03.838135 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:03.838119    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 04 23:38:03.838267 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:03.838253    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 04 23:38:03.848120 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:03.848103    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3463da-4922-4b81-a71e-8769f4a90494-metrics-certs\") pod \"network-metrics-daemon-rd7qq\" (UID: \"ca3463da-4922-4b81-a71e-8769f4a90494\") " pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:38:03.848886 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:03.848873    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 04 23:38:03.858772 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:03.858755    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh9p6\" (UniqueName: \"kubernetes.io/projected/3d5cbc4c-5f2e-4611-85ee-70e51ebadd34-kube-api-access-gh9p6\") pod \"network-check-target-ngp84\" (UID: \"3d5cbc4c-5f2e-4611-85ee-70e51ebadd34\") " pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:38:03.890003 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:03.889984    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8jjtb\""
Sep 04 23:38:03.892854 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:03.892843    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sj4n7"
Sep 04 23:38:03.898035 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:03.898022    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd7qq"
Sep 04 23:38:03.987925 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:03.987863    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vhhzf\""
Sep 04 23:38:03.994990 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:03.994974    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:38:04.168405 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:04.168375    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sj4n7"]
Sep 04 23:38:04.170648 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:38:04.170621    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc959dc_ab62_40f7_81ea_e62e4aafed21.slice/crio-9fd73aaf3980e0267514de8c1a0bf9fd37eabf2a8d8ca5e4b785e752fb80cf28 WatchSource:0}: Error finding container 9fd73aaf3980e0267514de8c1a0bf9fd37eabf2a8d8ca5e4b785e752fb80cf28: Status 404 returned error can't find the container with id 9fd73aaf3980e0267514de8c1a0bf9fd37eabf2a8d8ca5e4b785e752fb80cf28
Sep 04 23:38:04.173758 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:04.173738    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rd7qq"]
Sep 04 23:38:04.176479 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:38:04.176458    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3463da_4922_4b81_a71e_8769f4a90494.slice/crio-43e781b7c6b413a19240de4e7c4e9615a5cdc6257a142e938e2c53512e180bc2 WatchSource:0}: Error finding container 43e781b7c6b413a19240de4e7c4e9615a5cdc6257a142e938e2c53512e180bc2: Status 404 returned error can't find the container with id 43e781b7c6b413a19240de4e7c4e9615a5cdc6257a142e938e2c53512e180bc2
Sep 04 23:38:04.259654 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:04.259633    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ngp84"]
Sep 04 23:38:04.261432 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:38:04.261411    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5cbc4c_5f2e_4611_85ee_70e51ebadd34.slice/crio-f26ae6dff9d92c1e8c5a97c33b88b8dfae4e76d8774b9dba3111f9809b58d51f WatchSource:0}: Error finding container f26ae6dff9d92c1e8c5a97c33b88b8dfae4e76d8774b9dba3111f9809b58d51f: Status 404 returned error can't find the container with id f26ae6dff9d92c1e8c5a97c33b88b8dfae4e76d8774b9dba3111f9809b58d51f
Sep 04 23:38:04.330573 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:04.330549    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rd7qq" event={"ID":"ca3463da-4922-4b81-a71e-8769f4a90494","Type":"ContainerStarted","Data":"43e781b7c6b413a19240de4e7c4e9615a5cdc6257a142e938e2c53512e180bc2"}
Sep 04 23:38:04.331631 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:04.331613    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ngp84" event={"ID":"3d5cbc4c-5f2e-4611-85ee-70e51ebadd34","Type":"ContainerStarted","Data":"f26ae6dff9d92c1e8c5a97c33b88b8dfae4e76d8774b9dba3111f9809b58d51f"}
Sep 04 23:38:04.332640 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:04.332621    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sj4n7" event={"ID":"9dc959dc-ab62-40f7-81ea-e62e4aafed21","Type":"ContainerStarted","Data":"9fd73aaf3980e0267514de8c1a0bf9fd37eabf2a8d8ca5e4b785e752fb80cf28"}
Sep 04 23:38:06.042059 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.042021    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"]
Sep 04 23:38:06.048439 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.048405    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"
Sep 04 23:38:06.051452 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.051423    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 04 23:38:06.051789 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.051751    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 04 23:38:06.052146 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.052005    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-x56sz\""
Sep 04 23:38:06.053767 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.053113    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 04 23:38:06.053767 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.053320    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 04 23:38:06.053767 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.053526    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 04 23:38:06.054661 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.054628    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"]
Sep 04 23:38:06.062563 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.062541    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"]
Sep 04 23:38:06.065200 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.065181    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.067818 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.067798    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 04 23:38:06.067987 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.067966    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 04 23:38:06.068075 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.068041    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-w8sss\""
Sep 04 23:38:06.068446 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.068425    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 04 23:38:06.079422 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.079403    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"]
Sep 04 23:38:06.082050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.081891    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zf9mx"]
Sep 04 23:38:06.084142 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.084121    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.086902 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.086858    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 04 23:38:06.087018 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.086901    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 04 23:38:06.087389 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.087185    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pq79h\""
Sep 04 23:38:06.087389 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.087194    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 04 23:38:06.152833 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.152804    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fc699e03-b224-47ec-92dd-c1d4f41add41-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.153015 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.152851    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-textfile\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.153015 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.152878    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxjv2\" (UniqueName: \"kubernetes.io/projected/fc699e03-b224-47ec-92dd-c1d4f41add41-kube-api-access-pxjv2\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.153015 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.152901    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7mqw\" (UniqueName: \"kubernetes.io/projected/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-kube-api-access-m7mqw\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.153015 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.152963    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-tls\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.153252 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.153034    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-accelerators-collector-config\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.153252 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.153062    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc699e03-b224-47ec-92dd-c1d4f41add41-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.153252 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.153090    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9aa8ad72-aca0-4abb-9e56-efc967a6e1da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-z49js\" (UID: \"9aa8ad72-aca0-4abb-9e56-efc967a6e1da\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"
Sep 04 23:38:06.153252 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.153121    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-wtmp\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.153252 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.153159    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.153252 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.153189    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aa8ad72-aca0-4abb-9e56-efc967a6e1da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-z49js\" (UID: \"9aa8ad72-aca0-4abb-9e56-efc967a6e1da\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"
Sep 04 23:38:06.153518 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.153261    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc699e03-b224-47ec-92dd-c1d4f41add41-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.153518 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.153297    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-metrics-client-ca\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.153518 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.153322    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc699e03-b224-47ec-92dd-c1d4f41add41-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.153518 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.153361    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzb6w\" (UniqueName: \"kubernetes.io/projected/9aa8ad72-aca0-4abb-9e56-efc967a6e1da-kube-api-access-jzb6w\") pod \"openshift-state-metrics-657fd6c54-z49js\" (UID: \"9aa8ad72-aca0-4abb-9e56-efc967a6e1da\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"
Sep 04 23:38:06.153518 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.153402    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fc699e03-b224-47ec-92dd-c1d4f41add41-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.153518 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.153430    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-sys\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.153518 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.153459    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-root\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.153518 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.153494    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa8ad72-aca0-4abb-9e56-efc967a6e1da-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-z49js\" (UID: \"9aa8ad72-aca0-4abb-9e56-efc967a6e1da\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"
Sep 04 23:38:06.254142 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254112    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.254142 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254147    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aa8ad72-aca0-4abb-9e56-efc967a6e1da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-z49js\" (UID: \"9aa8ad72-aca0-4abb-9e56-efc967a6e1da\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"
Sep 04 23:38:06.254354 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254166    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc699e03-b224-47ec-92dd-c1d4f41add41-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.254354 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254183    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-metrics-client-ca\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.254354 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254205    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc699e03-b224-47ec-92dd-c1d4f41add41-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.254354 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:38:06.254300    2557 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 04 23:38:06.254547 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254349    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzb6w\" (UniqueName: \"kubernetes.io/projected/9aa8ad72-aca0-4abb-9e56-efc967a6e1da-kube-api-access-jzb6w\") pod \"openshift-state-metrics-657fd6c54-z49js\" (UID: \"9aa8ad72-aca0-4abb-9e56-efc967a6e1da\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"
Sep 04 23:38:06.254547 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:38:06.254366    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa8ad72-aca0-4abb-9e56-efc967a6e1da-openshift-state-metrics-tls podName:9aa8ad72-aca0-4abb-9e56-efc967a6e1da nodeName:}" failed. No retries permitted until 2026-09-04 23:38:06.754350408 +0000 UTC m=+69.172168730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9aa8ad72-aca0-4abb-9e56-efc967a6e1da-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-z49js" (UID: "9aa8ad72-aca0-4abb-9e56-efc967a6e1da") : secret "openshift-state-metrics-tls" not found
Sep 04 23:38:06.254547 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:38:06.254308    2557 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 04 23:38:06.254547 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254416    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fc699e03-b224-47ec-92dd-c1d4f41add41-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.254547 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:38:06.254437    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc699e03-b224-47ec-92dd-c1d4f41add41-kube-state-metrics-tls podName:fc699e03-b224-47ec-92dd-c1d4f41add41 nodeName:}" failed. No retries permitted until 2026-09-04 23:38:06.754422889 +0000 UTC m=+69.172241200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/fc699e03-b224-47ec-92dd-c1d4f41add41-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-hk5cq" (UID: "fc699e03-b224-47ec-92dd-c1d4f41add41") : secret "kube-state-metrics-tls" not found
Sep 04 23:38:06.254547 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254471    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-sys\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.254547 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254495    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-root\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.254547 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254526    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa8ad72-aca0-4abb-9e56-efc967a6e1da-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-z49js\" (UID: \"9aa8ad72-aca0-4abb-9e56-efc967a6e1da\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"
Sep 04 23:38:06.254547 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254551    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fc699e03-b224-47ec-92dd-c1d4f41add41-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.254992 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254588    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-textfile\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.254992 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254615    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxjv2\" (UniqueName: \"kubernetes.io/projected/fc699e03-b224-47ec-92dd-c1d4f41add41-kube-api-access-pxjv2\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.254992 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254639    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7mqw\" (UniqueName: \"kubernetes.io/projected/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-kube-api-access-m7mqw\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.254992 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254665    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-tls\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.254992 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254718    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-accelerators-collector-config\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.254992 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254747    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc699e03-b224-47ec-92dd-c1d4f41add41-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.254992 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254775    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9aa8ad72-aca0-4abb-9e56-efc967a6e1da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-z49js\" (UID: \"9aa8ad72-aca0-4abb-9e56-efc967a6e1da\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"
Sep 04 23:38:06.254992 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254812    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-wtmp\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.254992 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254914    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-metrics-client-ca\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.254992 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.254982    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc699e03-b224-47ec-92dd-c1d4f41add41-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.255489 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.255095    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fc699e03-b224-47ec-92dd-c1d4f41add41-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.255489 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.255130    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-sys\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.255489 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.255377    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-textfile\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.255489 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:38:06.255393    2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 04 23:38:06.255489 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.255408    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fc699e03-b224-47ec-92dd-c1d4f41add41-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.255489 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.255434    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-accelerators-collector-config\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.255489 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:38:06.255447    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-tls podName:9717ba4d-cf89-4e64-a7ce-4d119569e7f0 nodeName:}" failed. No retries permitted until 2026-09-04 23:38:06.755430339 +0000 UTC m=+69.173248652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-tls") pod "node-exporter-zf9mx" (UID: "9717ba4d-cf89-4e64-a7ce-4d119569e7f0") : secret "node-exporter-tls" not found
Sep 04 23:38:06.255489 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.255470    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-root\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.255875 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.255507    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-wtmp\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.256158 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.256136    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa8ad72-aca0-4abb-9e56-efc967a6e1da-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-z49js\" (UID: \"9aa8ad72-aca0-4abb-9e56-efc967a6e1da\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"
Sep 04 23:38:06.257035 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.257014    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.257559 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.257534    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc699e03-b224-47ec-92dd-c1d4f41add41-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.258089 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.258068    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9aa8ad72-aca0-4abb-9e56-efc967a6e1da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-z49js\" (UID: \"9aa8ad72-aca0-4abb-9e56-efc967a6e1da\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"
Sep 04 23:38:06.263559 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.263522    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxjv2\" (UniqueName: \"kubernetes.io/projected/fc699e03-b224-47ec-92dd-c1d4f41add41-kube-api-access-pxjv2\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.264383 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.264336    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzb6w\" (UniqueName: \"kubernetes.io/projected/9aa8ad72-aca0-4abb-9e56-efc967a6e1da-kube-api-access-jzb6w\") pod \"openshift-state-metrics-657fd6c54-z49js\" (UID: \"9aa8ad72-aca0-4abb-9e56-efc967a6e1da\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"
Sep 04 23:38:06.264472 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.264382    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7mqw\" (UniqueName: \"kubernetes.io/projected/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-kube-api-access-m7mqw\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.341994 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.341908    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rd7qq" event={"ID":"ca3463da-4922-4b81-a71e-8769f4a90494","Type":"ContainerStarted","Data":"67925f09ad964fc05374fe5a30590f29e33130dabfa57f1475afef5be59cfcb7"}
Sep 04 23:38:06.341994 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.341982    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rd7qq" event={"ID":"ca3463da-4922-4b81-a71e-8769f4a90494","Type":"ContainerStarted","Data":"42375a5f9cd95ee91c5f932f6656596916eb355da76bf810973c38b442b688e6"}
Sep 04 23:38:06.361043 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.360990    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rd7qq" podStartSLOduration=67.206343244 podStartE2EDuration="1m8.360970389s" totalImagesPullingTime="1.154627145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:38:06.360153666 +0000 UTC m=+68.777971989" watchObservedRunningTime="2026-09-04 23:38:06.360970389 +0000 UTC m=+68.778788708"
Sep 04 23:38:06.759367 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.759334    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-tls\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.759573 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.759430    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aa8ad72-aca0-4abb-9e56-efc967a6e1da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-z49js\" (UID: \"9aa8ad72-aca0-4abb-9e56-efc967a6e1da\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"
Sep 04 23:38:06.759573 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.759479    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc699e03-b224-47ec-92dd-c1d4f41add41-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.762202 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.762173    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc699e03-b224-47ec-92dd-c1d4f41add41-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-hk5cq\" (UID: \"fc699e03-b224-47ec-92dd-c1d4f41add41\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.762310 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.762175    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9717ba4d-cf89-4e64-a7ce-4d119569e7f0-node-exporter-tls\") pod \"node-exporter-zf9mx\" (UID: \"9717ba4d-cf89-4e64-a7ce-4d119569e7f0\") " pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:06.762310 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.762250    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aa8ad72-aca0-4abb-9e56-efc967a6e1da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-z49js\" (UID: \"9aa8ad72-aca0-4abb-9e56-efc967a6e1da\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"
Sep 04 23:38:06.961550 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.961516    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"
Sep 04 23:38:06.977775 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.977746    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"
Sep 04 23:38:06.994841 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:06.994818    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zf9mx"
Sep 04 23:38:08.802558 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:38:08.802514    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9717ba4d_cf89_4e64_a7ce_4d119569e7f0.slice/crio-bfd43abc07063b22c3f62c2586b8c4e99d2e4e5816316f80daab81e126b695ce WatchSource:0}: Error finding container bfd43abc07063b22c3f62c2586b8c4e99d2e4e5816316f80daab81e126b695ce: Status 404 returned error can't find the container with id bfd43abc07063b22c3f62c2586b8c4e99d2e4e5816316f80daab81e126b695ce
Sep 04 23:38:09.080161 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.080136    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-z49js"]
Sep 04 23:38:09.082684 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.082656    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq"]
Sep 04 23:38:09.083022 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:38:09.082928    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa8ad72_aca0_4abb_9e56_efc967a6e1da.slice/crio-8d92f1496efe58d169c6553c003a4d6d16718141c4be4b5b152d6f5b655d934b WatchSource:0}: Error finding container 8d92f1496efe58d169c6553c003a4d6d16718141c4be4b5b152d6f5b655d934b: Status 404 returned error can't find the container with id 8d92f1496efe58d169c6553c003a4d6d16718141c4be4b5b152d6f5b655d934b
Sep 04 23:38:09.085369 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:38:09.085344    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc699e03_b224_47ec_92dd_c1d4f41add41.slice/crio-2dfb3140daf012a84ba8837e2eaf9a4626cf39313ccd5a69fa37d3452f12d26c WatchSource:0}: Error finding container 2dfb3140daf012a84ba8837e2eaf9a4626cf39313ccd5a69fa37d3452f12d26c: Status 404 returned error can't find the container with id 2dfb3140daf012a84ba8837e2eaf9a4626cf39313ccd5a69fa37d3452f12d26c
Sep 04 23:38:09.098438 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.098412    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"]
Sep 04 23:38:09.101349 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.101333    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.104058 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.104038    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 04 23:38:09.104058 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.104048    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-thk7d\""
Sep 04 23:38:09.104170 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.104097    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9ivbjs645l8hm\""
Sep 04 23:38:09.104217 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.104200    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 04 23:38:09.104217 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.104210    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 04 23:38:09.104501 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.104481    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 04 23:38:09.104546 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.104505    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 04 23:38:09.113910 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.113889    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"]
Sep 04 23:38:09.178293 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.178265    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-grpc-tls\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.178394 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.178335    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-thanos-querier-tls\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.178394 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.178361    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbk44\" (UniqueName: \"kubernetes.io/projected/9bbca191-e2ec-4175-bc12-ebee023e8ba4-kube-api-access-dbk44\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.178394 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.178387    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.178553 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.178529    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.178602 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.178562    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.178602 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.178588    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.178672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.178613    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bbca191-e2ec-4175-bc12-ebee023e8ba4-metrics-client-ca\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.279832 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.279799    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.280009 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.279849    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.280009 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.279869    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.280009 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.279894    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bbca191-e2ec-4175-bc12-ebee023e8ba4-metrics-client-ca\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.280009 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.279931    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-grpc-tls\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.280009 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.279995    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-thanos-querier-tls\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.280271 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.280019    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbk44\" (UniqueName: \"kubernetes.io/projected/9bbca191-e2ec-4175-bc12-ebee023e8ba4-kube-api-access-dbk44\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.280271 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.280045    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.281152 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.281123    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bbca191-e2ec-4175-bc12-ebee023e8ba4-metrics-client-ca\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.282567 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.282545    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.282780 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.282758    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.282967 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.282923    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.283384 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.283359    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.283478 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.283388    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-grpc-tls\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.284402 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.284386    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9bbca191-e2ec-4175-bc12-ebee023e8ba4-secret-thanos-querier-tls\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.290336 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.290312    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbk44\" (UniqueName: \"kubernetes.io/projected/9bbca191-e2ec-4175-bc12-ebee023e8ba4-kube-api-access-dbk44\") pod \"thanos-querier-7fffc77b74-rjpc7\" (UID: \"9bbca191-e2ec-4175-bc12-ebee023e8ba4\") " pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.354261 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.354196    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq" event={"ID":"fc699e03-b224-47ec-92dd-c1d4f41add41","Type":"ContainerStarted","Data":"2dfb3140daf012a84ba8837e2eaf9a4626cf39313ccd5a69fa37d3452f12d26c"}
Sep 04 23:38:09.356319 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.356288    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ngp84" event={"ID":"3d5cbc4c-5f2e-4611-85ee-70e51ebadd34","Type":"ContainerStarted","Data":"01c95460ce1da6dc6e40f6bc612bc3db1896c9671cdffc1d28178fae5fa359a6"}
Sep 04 23:38:09.356424 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.356396    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:38:09.358237 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.358209    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sj4n7" event={"ID":"9dc959dc-ab62-40f7-81ea-e62e4aafed21","Type":"ContainerStarted","Data":"6d5d56075c0efacf73ff213f3addbd73cfdceced5eae2aa6281a9427a72b0abd"}
Sep 04 23:38:09.363884 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.363844    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf9mx" event={"ID":"9717ba4d-cf89-4e64-a7ce-4d119569e7f0","Type":"ContainerStarted","Data":"bfd43abc07063b22c3f62c2586b8c4e99d2e4e5816316f80daab81e126b695ce"}
Sep 04 23:38:09.366265 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.366245    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js" event={"ID":"9aa8ad72-aca0-4abb-9e56-efc967a6e1da","Type":"ContainerStarted","Data":"e0ff18f137e88cc9fa03d23ed581096f2a863f971f12d943ef5e2ad0f7723768"}
Sep 04 23:38:09.366352 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.366291    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js" event={"ID":"9aa8ad72-aca0-4abb-9e56-efc967a6e1da","Type":"ContainerStarted","Data":"69fc2241d1e4a7b8b7c85f98086c59b101c2bc5c06e052ff8229cd1114edafac"}
Sep 04 23:38:09.366352 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.366310    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js" event={"ID":"9aa8ad72-aca0-4abb-9e56-efc967a6e1da","Type":"ContainerStarted","Data":"8d92f1496efe58d169c6553c003a4d6d16718141c4be4b5b152d6f5b655d934b"}
Sep 04 23:38:09.371697 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.371647    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ngp84" podStartSLOduration=66.830625337 podStartE2EDuration="1m11.371630375s" totalImagesPullingTime="4.541005038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:38:09.371288882 +0000 UTC m=+71.789107201" watchObservedRunningTime="2026-09-04 23:38:09.371630375 +0000 UTC m=+71.789448696"
Sep 04 23:38:09.390726 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.390676    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sj4n7" podStartSLOduration=66.744118773 podStartE2EDuration="1m11.39066034s" totalImagesPullingTime="4.646541567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:38:09.390338651 +0000 UTC m=+71.808156969" watchObservedRunningTime="2026-09-04 23:38:09.39066034 +0000 UTC m=+71.808478659"
Sep 04 23:38:09.416576 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.416557    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:09.695296 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:09.695230    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"]
Sep 04 23:38:09.697971 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:38:09.697914    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbca191_e2ec_4175_bc12_ebee023e8ba4.slice/crio-c65264ef2866c4b3773007b40ead067f6caf9506097a4e9ed80e36fbf0ac2d5e WatchSource:0}: Error finding container c65264ef2866c4b3773007b40ead067f6caf9506097a4e9ed80e36fbf0ac2d5e: Status 404 returned error can't find the container with id c65264ef2866c4b3773007b40ead067f6caf9506097a4e9ed80e36fbf0ac2d5e
Sep 04 23:38:10.370967 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.370739    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-68cb67bcf5-j584m"]
Sep 04 23:38:10.374263 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.374184    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7" event={"ID":"9bbca191-e2ec-4175-bc12-ebee023e8ba4","Type":"ContainerStarted","Data":"c65264ef2866c4b3773007b40ead067f6caf9506097a4e9ed80e36fbf0ac2d5e"}
Sep 04 23:38:10.374396 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.374315    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.374515 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.374479    2557 generic.go:356] "Generic (PLEG): container finished" podID="9717ba4d-cf89-4e64-a7ce-4d119569e7f0" containerID="63772045edf1ef144a9307cb80661e491269d3041181052aa8d1d66accb00c63" exitCode=0
Sep 04 23:38:10.374669 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.374646    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf9mx" event={"ID":"9717ba4d-cf89-4e64-a7ce-4d119569e7f0","Type":"ContainerDied","Data":"63772045edf1ef144a9307cb80661e491269d3041181052aa8d1d66accb00c63"}
Sep 04 23:38:10.378056 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.378033    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 04 23:38:10.378160 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.378043    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 04 23:38:10.378469 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.378305    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 04 23:38:10.378469 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.378394    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ctdz8\""
Sep 04 23:38:10.378469 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.378432    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 04 23:38:10.378469 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.378449    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fnel22jcpqqvm\""
Sep 04 23:38:10.385826 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.385802    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-68cb67bcf5-j584m"]
Sep 04 23:38:10.488690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.488482    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88735e65-4065-495a-971e-e617a52e52de-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.488690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.488532    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/88735e65-4065-495a-971e-e617a52e52de-secret-metrics-server-tls\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.488690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.488593    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/88735e65-4065-495a-971e-e617a52e52de-audit-log\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.488690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.488678    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/88735e65-4065-495a-971e-e617a52e52de-metrics-server-audit-profiles\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.489111 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.488709    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc996\" (UniqueName: \"kubernetes.io/projected/88735e65-4065-495a-971e-e617a52e52de-kube-api-access-xc996\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.489111 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.488741    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/88735e65-4065-495a-971e-e617a52e52de-client-ca-bundle\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.489111 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.488812    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/88735e65-4065-495a-971e-e617a52e52de-secret-metrics-server-client-certs\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.590110 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.590082    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/88735e65-4065-495a-971e-e617a52e52de-client-ca-bundle\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.590268 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.590123    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/88735e65-4065-495a-971e-e617a52e52de-secret-metrics-server-client-certs\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.590268 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.590203    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88735e65-4065-495a-971e-e617a52e52de-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.590268 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.590235    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/88735e65-4065-495a-971e-e617a52e52de-secret-metrics-server-tls\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.590424 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.590273    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/88735e65-4065-495a-971e-e617a52e52de-audit-log\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.590424 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.590336    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/88735e65-4065-495a-971e-e617a52e52de-metrics-server-audit-profiles\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.590424 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.590367    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc996\" (UniqueName: \"kubernetes.io/projected/88735e65-4065-495a-971e-e617a52e52de-kube-api-access-xc996\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.590881 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.590822    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/88735e65-4065-495a-971e-e617a52e52de-audit-log\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.591031 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.590930    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88735e65-4065-495a-971e-e617a52e52de-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.591597 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.591570    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/88735e65-4065-495a-971e-e617a52e52de-metrics-server-audit-profiles\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.592958 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.592922    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/88735e65-4065-495a-971e-e617a52e52de-secret-metrics-server-tls\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.593059 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.593015    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/88735e65-4065-495a-971e-e617a52e52de-secret-metrics-server-client-certs\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.593126 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.593073    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/88735e65-4065-495a-971e-e617a52e52de-client-ca-bundle\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.598621 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.598597    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc996\" (UniqueName: \"kubernetes.io/projected/88735e65-4065-495a-971e-e617a52e52de-kube-api-access-xc996\") pod \"metrics-server-68cb67bcf5-j584m\" (UID: \"88735e65-4065-495a-971e-e617a52e52de\") " pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.685737 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.685644    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:10.691036 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.690999    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:38:10.691147 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.691092    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:38:10.691561 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.691541    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/514da723-8019-4825-a454-a1d56c30be6f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:38:10.693609 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.693588    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/514da723-8019-4825-a454-a1d56c30be6f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8th6\" (UID: \"514da723-8019-4825-a454-a1d56c30be6f\") " pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:38:10.804190 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.804158    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qcf5d\""
Sep 04 23:38:10.811851 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.811830    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l8th6"
Sep 04 23:38:10.843202 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.843174    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96"]
Sep 04 23:38:10.849251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.849135    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96"
Sep 04 23:38:10.852765 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.852175    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 04 23:38:10.852765 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.852196    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bgdll\""
Sep 04 23:38:10.855715 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.855690    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96"]
Sep 04 23:38:10.893170 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.893146    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25b90e0e-65d5-4510-a013-8c4795de5065-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8km96\" (UID: \"25b90e0e-65d5-4510-a013-8c4795de5065\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96"
Sep 04 23:38:10.893272 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.893181    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2sct\" (UniqueName: \"kubernetes.io/projected/25b90e0e-65d5-4510-a013-8c4795de5065-kube-api-access-w2sct\") pod \"monitoring-plugin-85bbbbbf4-8km96\" (UID: \"25b90e0e-65d5-4510-a013-8c4795de5065\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96"
Sep 04 23:38:10.995802 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.995187    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25b90e0e-65d5-4510-a013-8c4795de5065-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8km96\" (UID: \"25b90e0e-65d5-4510-a013-8c4795de5065\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96"
Sep 04 23:38:10.995802 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:10.995245    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2sct\" (UniqueName: \"kubernetes.io/projected/25b90e0e-65d5-4510-a013-8c4795de5065-kube-api-access-w2sct\") pod \"monitoring-plugin-85bbbbbf4-8km96\" (UID: \"25b90e0e-65d5-4510-a013-8c4795de5065\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96"
Sep 04 23:38:10.995802 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:38:10.995701    2557 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 04 23:38:10.995802 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:38:10.995769    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b90e0e-65d5-4510-a013-8c4795de5065-monitoring-plugin-cert podName:25b90e0e-65d5-4510-a013-8c4795de5065 nodeName:}" failed. No retries permitted until 2026-09-04 23:38:11.495748369 +0000 UTC m=+73.913566678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/25b90e0e-65d5-4510-a013-8c4795de5065-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-8km96" (UID: "25b90e0e-65d5-4510-a013-8c4795de5065") : secret "monitoring-plugin-cert" not found
Sep 04 23:38:11.008104 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.008076    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2sct\" (UniqueName: \"kubernetes.io/projected/25b90e0e-65d5-4510-a013-8c4795de5065-kube-api-access-w2sct\") pod \"monitoring-plugin-85bbbbbf4-8km96\" (UID: \"25b90e0e-65d5-4510-a013-8c4795de5065\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96"
Sep 04 23:38:11.097295 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.097260    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-68cb67bcf5-j584m"]
Sep 04 23:38:11.101753 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:38:11.101721    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88735e65_4065_495a_971e_e617a52e52de.slice/crio-407ae973ce298e660ddcad4c561e0e1660e988b0db5ba70bd8e48b45bccd7412 WatchSource:0}: Error finding container 407ae973ce298e660ddcad4c561e0e1660e988b0db5ba70bd8e48b45bccd7412: Status 404 returned error can't find the container with id 407ae973ce298e660ddcad4c561e0e1660e988b0db5ba70bd8e48b45bccd7412
Sep 04 23:38:11.107492 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.107413    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l8th6"]
Sep 04 23:38:11.111543 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:38:11.111511    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514da723_8019_4825_a454_a1d56c30be6f.slice/crio-ff45269c61e92a5fde9ec5d73cd73bf0ca66a7e9c3a75df572234171f6c9cccb WatchSource:0}: Error finding container ff45269c61e92a5fde9ec5d73cd73bf0ca66a7e9c3a75df572234171f6c9cccb: Status 404 returned error can't find the container with id ff45269c61e92a5fde9ec5d73cd73bf0ca66a7e9c3a75df572234171f6c9cccb
Sep 04 23:38:11.381776 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.381742    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq" event={"ID":"fc699e03-b224-47ec-92dd-c1d4f41add41","Type":"ContainerStarted","Data":"b20af3a1b5ce0ac5e607b5a1bab75f86fd18cb90b72c6ed814eefe60cd765566"}
Sep 04 23:38:11.382252 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.381823    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq" event={"ID":"fc699e03-b224-47ec-92dd-c1d4f41add41","Type":"ContainerStarted","Data":"88e0087defb7a739e06e539838f916bf47508c454d2ee1658883bc3694235e0d"}
Sep 04 23:38:11.382252 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.381856    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq" event={"ID":"fc699e03-b224-47ec-92dd-c1d4f41add41","Type":"ContainerStarted","Data":"c3853bcdd738747465a860d82bbecfdeeafb3d2ae38afe8a3ce5bfc79c9af8c2"}
Sep 04 23:38:11.383173 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.383148    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m" event={"ID":"88735e65-4065-495a-971e-e617a52e52de","Type":"ContainerStarted","Data":"407ae973ce298e660ddcad4c561e0e1660e988b0db5ba70bd8e48b45bccd7412"}
Sep 04 23:38:11.385642 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.385615    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf9mx" event={"ID":"9717ba4d-cf89-4e64-a7ce-4d119569e7f0","Type":"ContainerStarted","Data":"b628cd5c6e2350514b86937d7222145b367a252e2f15e20dce6e4078dd481389"}
Sep 04 23:38:11.385814 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.385670    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf9mx" event={"ID":"9717ba4d-cf89-4e64-a7ce-4d119569e7f0","Type":"ContainerStarted","Data":"881bc9ee854c5febcb3e63725fc92c8a6f7381020b24a606cbea73c273d2678c"}
Sep 04 23:38:11.387346 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.387322    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8th6" event={"ID":"514da723-8019-4825-a454-a1d56c30be6f","Type":"ContainerStarted","Data":"9fa4166c820c5ea4559faa75c2d66aefb0d5c4404906affdb988f2a939fb9ba0"}
Sep 04 23:38:11.387444 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.387372    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8th6" event={"ID":"514da723-8019-4825-a454-a1d56c30be6f","Type":"ContainerStarted","Data":"ff45269c61e92a5fde9ec5d73cd73bf0ca66a7e9c3a75df572234171f6c9cccb"}
Sep 04 23:38:11.389595 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.389543    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js" event={"ID":"9aa8ad72-aca0-4abb-9e56-efc967a6e1da","Type":"ContainerStarted","Data":"03516eed89236a5366ae907c1eaa4f4bea2cd83d4740e5504a4ad0bbc7a688c1"}
Sep 04 23:38:11.405438 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.405398    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hk5cq" podStartSLOduration=3.677662812 podStartE2EDuration="5.405380433s" totalImagesPullingTime="1.727717621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:38:11.403886038 +0000 UTC m=+73.821704406" watchObservedRunningTime="2026-09-04 23:38:11.405380433 +0000 UTC m=+73.823198762"
Sep 04 23:38:11.436340 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.436294    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-z49js" podStartSLOduration=3.815918014 podStartE2EDuration="5.436259744s" totalImagesPullingTime="1.62034173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:38:11.435287331 +0000 UTC m=+73.853105662" watchObservedRunningTime="2026-09-04 23:38:11.436259744 +0000 UTC m=+73.854078063"
Sep 04 23:38:11.470406 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.470368    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zf9mx" podStartSLOduration=4.504062715 podStartE2EDuration="5.470358144s" totalImagesPullingTime="896.073595ms" totalInitContainerRuntime="70.221834ms" isStatefulPod=true podCreationTimestamp="2026-09-04 23:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:38:11.469156133 +0000 UTC m=+73.886974463" watchObservedRunningTime="2026-09-04 23:38:11.470358144 +0000 UTC m=+73.888176462"
Sep 04 23:38:11.498739 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.498709    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25b90e0e-65d5-4510-a013-8c4795de5065-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8km96\" (UID: \"25b90e0e-65d5-4510-a013-8c4795de5065\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96"
Sep 04 23:38:11.501484 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.501464    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25b90e0e-65d5-4510-a013-8c4795de5065-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8km96\" (UID: \"25b90e0e-65d5-4510-a013-8c4795de5065\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96"
Sep 04 23:38:11.505337 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:11.505319    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96"
Sep 04 23:38:12.109680 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:12.109652    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96"]
Sep 04 23:38:12.113029 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:38:12.112968    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b90e0e_65d5_4510_a013_8c4795de5065.slice/crio-2c366f77283a29ef91927b39ee94a99b0780fe63653f650443a7c3df1f0bfc21 WatchSource:0}: Error finding container 2c366f77283a29ef91927b39ee94a99b0780fe63653f650443a7c3df1f0bfc21: Status 404 returned error can't find the container with id 2c366f77283a29ef91927b39ee94a99b0780fe63653f650443a7c3df1f0bfc21
Sep 04 23:38:12.396292 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:12.396205    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7" event={"ID":"9bbca191-e2ec-4175-bc12-ebee023e8ba4","Type":"ContainerStarted","Data":"125cd067f2d805b2756211b1fdcb0e70bdae783a6380d01de3a3bc2777581a4c"}
Sep 04 23:38:12.396674 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:12.396290    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7" event={"ID":"9bbca191-e2ec-4175-bc12-ebee023e8ba4","Type":"ContainerStarted","Data":"b44dbb4fb227b9b3461782fff9af0996b63e110357d5d79144ee265e994b37da"}
Sep 04 23:38:12.396674 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:12.396336    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7" event={"ID":"9bbca191-e2ec-4175-bc12-ebee023e8ba4","Type":"ContainerStarted","Data":"c2b7e6c2e1ff3b4498333d3bbbdc1d05aee8ab70343a51f12dc2b53ba2cc96b5"}
Sep 04 23:38:12.397690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:12.397661    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96" event={"ID":"25b90e0e-65d5-4510-a013-8c4795de5065","Type":"ContainerStarted","Data":"2c366f77283a29ef91927b39ee94a99b0780fe63653f650443a7c3df1f0bfc21"}
Sep 04 23:38:12.399765 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:12.399719    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8th6" event={"ID":"514da723-8019-4825-a454-a1d56c30be6f","Type":"ContainerStarted","Data":"c9a91ea3b9f2bb868cf6a32bcb0d19e1edbc9955008ca51b5412727e9d48db03"}
Sep 04 23:38:13.408700 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:13.408626    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m" event={"ID":"88735e65-4065-495a-971e-e617a52e52de","Type":"ContainerStarted","Data":"5a378006df465c614af0062b2b10e7f08eea3aa174f889d7694c63a861f4bd39"}
Sep 04 23:38:13.428341 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:13.428289    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m" podStartSLOduration=1.5726026100000001 podStartE2EDuration="3.42826794s" totalImagesPullingTime="1.85566533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:38:13.425689542 +0000 UTC m=+75.843507873" watchObservedRunningTime="2026-09-04 23:38:13.42826794 +0000 UTC m=+75.846086260"
Sep 04 23:38:14.415127 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:14.415059    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7" event={"ID":"9bbca191-e2ec-4175-bc12-ebee023e8ba4","Type":"ContainerStarted","Data":"efabd9377050308f0f6bf122d836ac7258068b92af62c7c4dcc239967cd0b8a3"}
Sep 04 23:38:14.415127 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:14.415122    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7" event={"ID":"9bbca191-e2ec-4175-bc12-ebee023e8ba4","Type":"ContainerStarted","Data":"cb4c5d64365c63f74b8d855ee7a06e9001eaec1d87c10d34313c41f609cc2322"}
Sep 04 23:38:15.422009 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:15.421975    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7" event={"ID":"9bbca191-e2ec-4175-bc12-ebee023e8ba4","Type":"ContainerStarted","Data":"b1eb862773dcabc5a7f406364292ebbec1287609c2b201f2457810266a889fa1"}
Sep 04 23:38:15.422432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:15.422106    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:15.423630 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:15.423599    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96" event={"ID":"25b90e0e-65d5-4510-a013-8c4795de5065","Type":"ContainerStarted","Data":"49f73f239be14875c51f7b032acf1ea11fcbdfdf583f40d8ef9e38c532d9b526"}
Sep 04 23:38:15.423764 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:15.423754    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96"
Sep 04 23:38:15.427883 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:15.427857    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8th6" event={"ID":"514da723-8019-4825-a454-a1d56c30be6f","Type":"ContainerStarted","Data":"6cc22e21845bb0aa2ea1ed1cdc27076d9f72d9d1287493c8553230243288ebac"}
Sep 04 23:38:15.430209 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:15.430190    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96"
Sep 04 23:38:15.444948 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:15.444903    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7" podStartSLOduration=3.022795459 podStartE2EDuration="6.444889614s" totalImagesPullingTime="3.422094155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:38:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:38:15.443192521 +0000 UTC m=+77.861010838" watchObservedRunningTime="2026-09-04 23:38:15.444889614 +0000 UTC m=+77.862707932"
Sep 04 23:38:15.463672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:15.463629    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l8th6" podStartSLOduration=18.118326993 podStartE2EDuration="21.46361724s" totalImagesPullingTime="3.345290247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:37:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:38:15.462175096 +0000 UTC m=+77.879993448" watchObservedRunningTime="2026-09-04 23:38:15.46361724 +0000 UTC m=+77.881435557"
Sep 04 23:38:15.476084 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:15.476039    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8km96" podStartSLOduration=3.005003671 podStartE2EDuration="5.476027188s" totalImagesPullingTime="2.471023517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:38:15.475211273 +0000 UTC m=+77.893029592" watchObservedRunningTime="2026-09-04 23:38:15.476027188 +0000 UTC m=+77.893845505"
Sep 04 23:38:21.437968 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:21.437834    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7fffc77b74-rjpc7"
Sep 04 23:38:30.686002 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:30.685973    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:30.686504 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:30.686032    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:40.377148 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:40.377114    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ngp84"
Sep 04 23:38:50.691312 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:50.691285    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:38:50.694982 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:38:50.694963    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-68cb67bcf5-j584m"
Sep 04 23:40:20.781319 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:20.781229    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"]
Sep 04 23:40:20.783976 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:20.783956    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"
Sep 04 23:40:20.786713 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:20.786692    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 04 23:40:20.786840 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:20.786802    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 04 23:40:20.786960 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:20.786922    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l949n\""
Sep 04 23:40:20.792680 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:20.792655    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"]
Sep 04 23:40:20.843988 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:20.843963    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck2lq\" (UniqueName: \"kubernetes.io/projected/dfa13793-4e57-4d53-806c-ac2b0f3b9768-kube-api-access-ck2lq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k\" (UID: \"dfa13793-4e57-4d53-806c-ac2b0f3b9768\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"
Sep 04 23:40:20.844090 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:20.844017    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dfa13793-4e57-4d53-806c-ac2b0f3b9768-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k\" (UID: \"dfa13793-4e57-4d53-806c-ac2b0f3b9768\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"
Sep 04 23:40:20.844090 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:20.844039    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dfa13793-4e57-4d53-806c-ac2b0f3b9768-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k\" (UID: \"dfa13793-4e57-4d53-806c-ac2b0f3b9768\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"
Sep 04 23:40:20.944847 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:20.944828    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck2lq\" (UniqueName: \"kubernetes.io/projected/dfa13793-4e57-4d53-806c-ac2b0f3b9768-kube-api-access-ck2lq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k\" (UID: \"dfa13793-4e57-4d53-806c-ac2b0f3b9768\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"
Sep 04 23:40:20.944987 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:20.944892    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dfa13793-4e57-4d53-806c-ac2b0f3b9768-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k\" (UID: \"dfa13793-4e57-4d53-806c-ac2b0f3b9768\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"
Sep 04 23:40:20.944987 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:20.944926    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dfa13793-4e57-4d53-806c-ac2b0f3b9768-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k\" (UID: \"dfa13793-4e57-4d53-806c-ac2b0f3b9768\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"
Sep 04 23:40:20.945224 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:20.945203    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dfa13793-4e57-4d53-806c-ac2b0f3b9768-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k\" (UID: \"dfa13793-4e57-4d53-806c-ac2b0f3b9768\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"
Sep 04 23:40:20.945290 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:20.945244    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dfa13793-4e57-4d53-806c-ac2b0f3b9768-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k\" (UID: \"dfa13793-4e57-4d53-806c-ac2b0f3b9768\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"
Sep 04 23:40:20.953086 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:20.953067    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck2lq\" (UniqueName: \"kubernetes.io/projected/dfa13793-4e57-4d53-806c-ac2b0f3b9768-kube-api-access-ck2lq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k\" (UID: \"dfa13793-4e57-4d53-806c-ac2b0f3b9768\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"
Sep 04 23:40:21.093461 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:21.093404    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"
Sep 04 23:40:21.358524 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:21.358501    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"]
Sep 04 23:40:21.360962 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:40:21.360926    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa13793_4e57_4d53_806c_ac2b0f3b9768.slice/crio-31ece249c4695a45ef3ba71c97d1ae02ba692f5b1de40c9aecc1022708f35458 WatchSource:0}: Error finding container 31ece249c4695a45ef3ba71c97d1ae02ba692f5b1de40c9aecc1022708f35458: Status 404 returned error can't find the container with id 31ece249c4695a45ef3ba71c97d1ae02ba692f5b1de40c9aecc1022708f35458
Sep 04 23:40:21.862658 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:21.862626    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k" event={"ID":"dfa13793-4e57-4d53-806c-ac2b0f3b9768","Type":"ContainerStarted","Data":"31ece249c4695a45ef3ba71c97d1ae02ba692f5b1de40c9aecc1022708f35458"}
Sep 04 23:40:27.889169 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:27.889143    2557 generic.go:356] "Generic (PLEG): container finished" podID="dfa13793-4e57-4d53-806c-ac2b0f3b9768" containerID="09e170cd321fcd3cc757a6e8da03db10b79340efdd55810df842096a97d7181d" exitCode=0
Sep 04 23:40:27.889490 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:27.889181    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k" event={"ID":"dfa13793-4e57-4d53-806c-ac2b0f3b9768","Type":"ContainerDied","Data":"09e170cd321fcd3cc757a6e8da03db10b79340efdd55810df842096a97d7181d"}
Sep 04 23:40:36.922923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:36.922889    2557 generic.go:356] "Generic (PLEG): container finished" podID="dfa13793-4e57-4d53-806c-ac2b0f3b9768" containerID="d49c0cdc6636729b3f32de09e4b34ff981c2090a005f5b6712817a8a45c47c71" exitCode=0
Sep 04 23:40:36.923310 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:36.922971    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k" event={"ID":"dfa13793-4e57-4d53-806c-ac2b0f3b9768","Type":"ContainerDied","Data":"d49c0cdc6636729b3f32de09e4b34ff981c2090a005f5b6712817a8a45c47c71"}
Sep 04 23:40:43.951644 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:43.951611    2557 generic.go:356] "Generic (PLEG): container finished" podID="dfa13793-4e57-4d53-806c-ac2b0f3b9768" containerID="1a53c200f0105bf567dda9fd20f5504d8d2a77b187a638faa73fc4d7bbd2e5fb" exitCode=0
Sep 04 23:40:43.952026 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:43.951651    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k" event={"ID":"dfa13793-4e57-4d53-806c-ac2b0f3b9768","Type":"ContainerDied","Data":"1a53c200f0105bf567dda9fd20f5504d8d2a77b187a638faa73fc4d7bbd2e5fb"}
Sep 04 23:40:45.079063 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:45.079042    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"
Sep 04 23:40:45.229569 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:45.229486    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dfa13793-4e57-4d53-806c-ac2b0f3b9768-util\") pod \"dfa13793-4e57-4d53-806c-ac2b0f3b9768\" (UID: \"dfa13793-4e57-4d53-806c-ac2b0f3b9768\") "
Sep 04 23:40:45.229569 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:45.229563    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dfa13793-4e57-4d53-806c-ac2b0f3b9768-bundle\") pod \"dfa13793-4e57-4d53-806c-ac2b0f3b9768\" (UID: \"dfa13793-4e57-4d53-806c-ac2b0f3b9768\") "
Sep 04 23:40:45.229771 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:45.229585    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck2lq\" (UniqueName: \"kubernetes.io/projected/dfa13793-4e57-4d53-806c-ac2b0f3b9768-kube-api-access-ck2lq\") pod \"dfa13793-4e57-4d53-806c-ac2b0f3b9768\" (UID: \"dfa13793-4e57-4d53-806c-ac2b0f3b9768\") "
Sep 04 23:40:45.230394 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:45.230362    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa13793-4e57-4d53-806c-ac2b0f3b9768-bundle" (OuterVolumeSpecName: "bundle") pod "dfa13793-4e57-4d53-806c-ac2b0f3b9768" (UID: "dfa13793-4e57-4d53-806c-ac2b0f3b9768"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:40:45.231636 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:45.231606    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa13793-4e57-4d53-806c-ac2b0f3b9768-kube-api-access-ck2lq" (OuterVolumeSpecName: "kube-api-access-ck2lq") pod "dfa13793-4e57-4d53-806c-ac2b0f3b9768" (UID: "dfa13793-4e57-4d53-806c-ac2b0f3b9768"). InnerVolumeSpecName "kube-api-access-ck2lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:40:45.234784 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:45.234754    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa13793-4e57-4d53-806c-ac2b0f3b9768-util" (OuterVolumeSpecName: "util") pod "dfa13793-4e57-4d53-806c-ac2b0f3b9768" (UID: "dfa13793-4e57-4d53-806c-ac2b0f3b9768"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:40:45.330410 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:45.330377    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dfa13793-4e57-4d53-806c-ac2b0f3b9768-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:40:45.330410 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:45.330408    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck2lq\" (UniqueName: \"kubernetes.io/projected/dfa13793-4e57-4d53-806c-ac2b0f3b9768-kube-api-access-ck2lq\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:40:45.330410 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:45.330417    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dfa13793-4e57-4d53-806c-ac2b0f3b9768-util\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:40:45.960801 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:45.960773    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k" event={"ID":"dfa13793-4e57-4d53-806c-ac2b0f3b9768","Type":"ContainerDied","Data":"31ece249c4695a45ef3ba71c97d1ae02ba692f5b1de40c9aecc1022708f35458"}
Sep 04 23:40:45.960934 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:45.960817    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31ece249c4695a45ef3ba71c97d1ae02ba692f5b1de40c9aecc1022708f35458"
Sep 04 23:40:45.960934 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:45.960786    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dp22k"
Sep 04 23:40:53.980696 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:53.980668    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nnx9z"]
Sep 04 23:40:53.981105 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:53.980998    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa13793-4e57-4d53-806c-ac2b0f3b9768" containerName="pull"
Sep 04 23:40:53.981105 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:53.981011    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa13793-4e57-4d53-806c-ac2b0f3b9768" containerName="pull"
Sep 04 23:40:53.981105 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:53.981020    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa13793-4e57-4d53-806c-ac2b0f3b9768" containerName="extract"
Sep 04 23:40:53.981105 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:53.981026    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa13793-4e57-4d53-806c-ac2b0f3b9768" containerName="extract"
Sep 04 23:40:53.981105 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:53.981036    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa13793-4e57-4d53-806c-ac2b0f3b9768" containerName="util"
Sep 04 23:40:53.981105 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:53.981041    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa13793-4e57-4d53-806c-ac2b0f3b9768" containerName="util"
Sep 04 23:40:53.981105 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:53.981094    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfa13793-4e57-4d53-806c-ac2b0f3b9768" containerName="extract"
Sep 04 23:40:53.983812 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:53.983797    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nnx9z"
Sep 04 23:40:53.986526 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:53.986495    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fhj5b\""
Sep 04 23:40:53.986526 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:53.986509    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 04 23:40:53.986731 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:53.986546    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 04 23:40:53.994692 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:53.994671    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nnx9z"]
Sep 04 23:40:54.086429 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:54.086404    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c531847-406b-468a-a678-60d53d81825a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nnx9z\" (UID: \"1c531847-406b-468a-a678-60d53d81825a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nnx9z"
Sep 04 23:40:54.086535 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:54.086458    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2drtx\" (UniqueName: \"kubernetes.io/projected/1c531847-406b-468a-a678-60d53d81825a-kube-api-access-2drtx\") pod \"cert-manager-operator-controller-manager-55c444bf86-nnx9z\" (UID: \"1c531847-406b-468a-a678-60d53d81825a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nnx9z"
Sep 04 23:40:54.187515 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:54.187492    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c531847-406b-468a-a678-60d53d81825a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nnx9z\" (UID: \"1c531847-406b-468a-a678-60d53d81825a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nnx9z"
Sep 04 23:40:54.187613 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:54.187534    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2drtx\" (UniqueName: \"kubernetes.io/projected/1c531847-406b-468a-a678-60d53d81825a-kube-api-access-2drtx\") pod \"cert-manager-operator-controller-manager-55c444bf86-nnx9z\" (UID: \"1c531847-406b-468a-a678-60d53d81825a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nnx9z"
Sep 04 23:40:54.187831 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:54.187814    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c531847-406b-468a-a678-60d53d81825a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nnx9z\" (UID: \"1c531847-406b-468a-a678-60d53d81825a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nnx9z"
Sep 04 23:40:54.195183 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:54.195167    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2drtx\" (UniqueName: \"kubernetes.io/projected/1c531847-406b-468a-a678-60d53d81825a-kube-api-access-2drtx\") pod \"cert-manager-operator-controller-manager-55c444bf86-nnx9z\" (UID: \"1c531847-406b-468a-a678-60d53d81825a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nnx9z"
Sep 04 23:40:54.293845 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:54.293791    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nnx9z"
Sep 04 23:40:54.568740 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:54.568670    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nnx9z"]
Sep 04 23:40:54.572172 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:40:54.572141    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c531847_406b_468a_a678_60d53d81825a.slice/crio-60b89b878fa7935881a20d6a31f57d7f8c61b56e92caacf87e314689e102086c WatchSource:0}: Error finding container 60b89b878fa7935881a20d6a31f57d7f8c61b56e92caacf87e314689e102086c: Status 404 returned error can't find the container with id 60b89b878fa7935881a20d6a31f57d7f8c61b56e92caacf87e314689e102086c
Sep 04 23:40:54.995992 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:54.995966    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nnx9z" event={"ID":"1c531847-406b-468a-a678-60d53d81825a","Type":"ContainerStarted","Data":"60b89b878fa7935881a20d6a31f57d7f8c61b56e92caacf87e314689e102086c"}
Sep 04 23:40:57.007186 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.007103    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nnx9z" event={"ID":"1c531847-406b-468a-a678-60d53d81825a","Type":"ContainerStarted","Data":"9463e6c91f9ee8f1c50d1533bc40414e22e29289a3709d3f01410b8849dbc3da"}
Sep 04 23:40:57.029797 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.029743    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nnx9z" podStartSLOduration=1.887989445 podStartE2EDuration="4.029724191s" totalImagesPullingTime="2.141734746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:40:57.026546063 +0000 UTC m=+239.444364382" watchObservedRunningTime="2026-09-04 23:40:57.029724191 +0000 UTC m=+239.447542510"
Sep 04 23:40:57.594119 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.594079    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"]
Sep 04 23:40:57.597831 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.597811    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"
Sep 04 23:40:57.601214 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.601184    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 04 23:40:57.602075 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.602058    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l949n\""
Sep 04 23:40:57.602226 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.602209    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 04 23:40:57.616700 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.616680    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"]
Sep 04 23:40:57.715533 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.715495    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86ab9f3c-72be-4248-a3ba-f595a6a4723e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx\" (UID: \"86ab9f3c-72be-4248-a3ba-f595a6a4723e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"
Sep 04 23:40:57.715665 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.715543    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86ab9f3c-72be-4248-a3ba-f595a6a4723e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx\" (UID: \"86ab9f3c-72be-4248-a3ba-f595a6a4723e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"
Sep 04 23:40:57.715738 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.715660    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgpcx\" (UniqueName: \"kubernetes.io/projected/86ab9f3c-72be-4248-a3ba-f595a6a4723e-kube-api-access-jgpcx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx\" (UID: \"86ab9f3c-72be-4248-a3ba-f595a6a4723e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"
Sep 04 23:40:57.815059 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.815030    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5h4fw"]
Sep 04 23:40:57.816494 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.816473    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86ab9f3c-72be-4248-a3ba-f595a6a4723e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx\" (UID: \"86ab9f3c-72be-4248-a3ba-f595a6a4723e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"
Sep 04 23:40:57.816597 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.816501    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86ab9f3c-72be-4248-a3ba-f595a6a4723e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx\" (UID: \"86ab9f3c-72be-4248-a3ba-f595a6a4723e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"
Sep 04 23:40:57.816597 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.816540    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgpcx\" (UniqueName: \"kubernetes.io/projected/86ab9f3c-72be-4248-a3ba-f595a6a4723e-kube-api-access-jgpcx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx\" (UID: \"86ab9f3c-72be-4248-a3ba-f595a6a4723e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"
Sep 04 23:40:57.816838 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.816821    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86ab9f3c-72be-4248-a3ba-f595a6a4723e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx\" (UID: \"86ab9f3c-72be-4248-a3ba-f595a6a4723e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"
Sep 04 23:40:57.816901 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.816854    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86ab9f3c-72be-4248-a3ba-f595a6a4723e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx\" (UID: \"86ab9f3c-72be-4248-a3ba-f595a6a4723e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"
Sep 04 23:40:57.818277 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.818262    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5h4fw"
Sep 04 23:40:57.820578 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.820552    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 04 23:40:57.820578 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.820572    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 04 23:40:57.820705 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.820556    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-j7vqr\""
Sep 04 23:40:57.827026 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.827007    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5h4fw"]
Sep 04 23:40:57.829440 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.829416    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgpcx\" (UniqueName: \"kubernetes.io/projected/86ab9f3c-72be-4248-a3ba-f595a6a4723e-kube-api-access-jgpcx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx\" (UID: \"86ab9f3c-72be-4248-a3ba-f595a6a4723e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"
Sep 04 23:40:57.906799 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.906735    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"
Sep 04 23:40:57.916999 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.916978    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4g8d\" (UniqueName: \"kubernetes.io/projected/6b1f53f6-1c95-4f7a-832a-a0cd1556fcee-kube-api-access-g4g8d\") pod \"cert-manager-webhook-6486ffd997-5h4fw\" (UID: \"6b1f53f6-1c95-4f7a-832a-a0cd1556fcee\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5h4fw"
Sep 04 23:40:57.917069 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:57.917010    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b1f53f6-1c95-4f7a-832a-a0cd1556fcee-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5h4fw\" (UID: \"6b1f53f6-1c95-4f7a-832a-a0cd1556fcee\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5h4fw"
Sep 04 23:40:58.017908 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:58.017879    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4g8d\" (UniqueName: \"kubernetes.io/projected/6b1f53f6-1c95-4f7a-832a-a0cd1556fcee-kube-api-access-g4g8d\") pod \"cert-manager-webhook-6486ffd997-5h4fw\" (UID: \"6b1f53f6-1c95-4f7a-832a-a0cd1556fcee\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5h4fw"
Sep 04 23:40:58.017908 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:58.017910    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b1f53f6-1c95-4f7a-832a-a0cd1556fcee-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5h4fw\" (UID: \"6b1f53f6-1c95-4f7a-832a-a0cd1556fcee\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5h4fw"
Sep 04 23:40:58.026250 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:58.026223    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 04 23:40:58.026250 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:58.026228    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b1f53f6-1c95-4f7a-832a-a0cd1556fcee-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5h4fw\" (UID: \"6b1f53f6-1c95-4f7a-832a-a0cd1556fcee\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5h4fw"
Sep 04 23:40:58.036863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:58.036840    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 04 23:40:58.047231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:58.047211    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4g8d\" (UniqueName: \"kubernetes.io/projected/6b1f53f6-1c95-4f7a-832a-a0cd1556fcee-kube-api-access-g4g8d\") pod \"cert-manager-webhook-6486ffd997-5h4fw\" (UID: \"6b1f53f6-1c95-4f7a-832a-a0cd1556fcee\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5h4fw"
Sep 04 23:40:58.136041 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:58.136021    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-j7vqr\""
Sep 04 23:40:58.144146 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:58.144128    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5h4fw"
Sep 04 23:40:58.184808 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:58.184104    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"]
Sep 04 23:40:58.186770 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:40:58.186744    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ab9f3c_72be_4248_a3ba_f595a6a4723e.slice/crio-412cc005f2f0abe044216953e7387cba01fb1994c0d71c481193fbfe9559d526 WatchSource:0}: Error finding container 412cc005f2f0abe044216953e7387cba01fb1994c0d71c481193fbfe9559d526: Status 404 returned error can't find the container with id 412cc005f2f0abe044216953e7387cba01fb1994c0d71c481193fbfe9559d526
Sep 04 23:40:58.420368 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:58.420302    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5h4fw"]
Sep 04 23:40:58.423167 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:40:58.423145    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b1f53f6_1c95_4f7a_832a_a0cd1556fcee.slice/crio-8ee9e493c854e6938556a33f69f92386f2fb3941c58acc47c4be1ddda8348125 WatchSource:0}: Error finding container 8ee9e493c854e6938556a33f69f92386f2fb3941c58acc47c4be1ddda8348125: Status 404 returned error can't find the container with id 8ee9e493c854e6938556a33f69f92386f2fb3941c58acc47c4be1ddda8348125
Sep 04 23:40:59.016787 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:59.016751    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5h4fw" event={"ID":"6b1f53f6-1c95-4f7a-832a-a0cd1556fcee","Type":"ContainerStarted","Data":"8ee9e493c854e6938556a33f69f92386f2fb3941c58acc47c4be1ddda8348125"}
Sep 04 23:40:59.018127 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:59.018100    2557 generic.go:356] "Generic (PLEG): container finished" podID="86ab9f3c-72be-4248-a3ba-f595a6a4723e" containerID="be2c375b099c1e67cd88e350b52f21e7084b011ef66d5c9c0a1e8303bb2f410e" exitCode=0
Sep 04 23:40:59.018518 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:59.018189    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx" event={"ID":"86ab9f3c-72be-4248-a3ba-f595a6a4723e","Type":"ContainerDied","Data":"be2c375b099c1e67cd88e350b52f21e7084b011ef66d5c9c0a1e8303bb2f410e"}
Sep 04 23:40:59.018518 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:40:59.018262    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx" event={"ID":"86ab9f3c-72be-4248-a3ba-f595a6a4723e","Type":"ContainerStarted","Data":"412cc005f2f0abe044216953e7387cba01fb1994c0d71c481193fbfe9559d526"}
Sep 04 23:41:01.371356 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:01.371321    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-f6mmm"]
Sep 04 23:41:01.376832 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:01.376807    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-f6mmm"
Sep 04 23:41:01.379506 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:01.379483    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8z7mj\""
Sep 04 23:41:01.385017 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:01.384998    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-f6mmm"]
Sep 04 23:41:01.447156 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:01.447118    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/836a759d-06ff-42c5-a5d3-70e88815ddd3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-f6mmm\" (UID: \"836a759d-06ff-42c5-a5d3-70e88815ddd3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f6mmm"
Sep 04 23:41:01.447308 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:01.447221    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqc7z\" (UniqueName: \"kubernetes.io/projected/836a759d-06ff-42c5-a5d3-70e88815ddd3-kube-api-access-bqc7z\") pod \"cert-manager-cainjector-559cdd8c87-f6mmm\" (UID: \"836a759d-06ff-42c5-a5d3-70e88815ddd3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f6mmm"
Sep 04 23:41:01.547834 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:01.547801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqc7z\" (UniqueName: \"kubernetes.io/projected/836a759d-06ff-42c5-a5d3-70e88815ddd3-kube-api-access-bqc7z\") pod \"cert-manager-cainjector-559cdd8c87-f6mmm\" (UID: \"836a759d-06ff-42c5-a5d3-70e88815ddd3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f6mmm"
Sep 04 23:41:01.548027 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:01.547879    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/836a759d-06ff-42c5-a5d3-70e88815ddd3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-f6mmm\" (UID: \"836a759d-06ff-42c5-a5d3-70e88815ddd3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f6mmm"
Sep 04 23:41:01.556338 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:01.556306    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/836a759d-06ff-42c5-a5d3-70e88815ddd3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-f6mmm\" (UID: \"836a759d-06ff-42c5-a5d3-70e88815ddd3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f6mmm"
Sep 04 23:41:01.556496 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:01.556476    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqc7z\" (UniqueName: \"kubernetes.io/projected/836a759d-06ff-42c5-a5d3-70e88815ddd3-kube-api-access-bqc7z\") pod \"cert-manager-cainjector-559cdd8c87-f6mmm\" (UID: \"836a759d-06ff-42c5-a5d3-70e88815ddd3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f6mmm"
Sep 04 23:41:01.688953 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:01.688873    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-f6mmm"
Sep 04 23:41:02.036289 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:02.036067    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5h4fw" event={"ID":"6b1f53f6-1c95-4f7a-832a-a0cd1556fcee","Type":"ContainerStarted","Data":"cfe677b61151efe87111a716d8f590a1f888bbf3a6162dce8fab35233827dbec"}
Sep 04 23:41:02.036289 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:02.036117    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5h4fw"
Sep 04 23:41:02.038069 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:02.038045    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx" event={"ID":"86ab9f3c-72be-4248-a3ba-f595a6a4723e","Type":"ContainerStarted","Data":"19a5e339ea3a761a22816ec688a33aa4be5a6f21c8f7aeed85dc054c04f205d5"}
Sep 04 23:41:02.051924 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:02.051885    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5h4fw" podStartSLOduration=1.586836247 podStartE2EDuration="5.051871176s" totalImagesPullingTime="3.465034929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:41:02.050249694 +0000 UTC m=+244.468068041" watchObservedRunningTime="2026-09-04 23:41:02.051871176 +0000 UTC m=+244.469689494"
Sep 04 23:41:02.149357 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:02.149219    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-f6mmm"]
Sep 04 23:41:02.151275 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:41:02.151250    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod836a759d_06ff_42c5_a5d3_70e88815ddd3.slice/crio-9c70a35247a90ad07de6fb0cac2292be4f952337f98d2a2ac0f46b1aebbce033 WatchSource:0}: Error finding container 9c70a35247a90ad07de6fb0cac2292be4f952337f98d2a2ac0f46b1aebbce033: Status 404 returned error can't find the container with id 9c70a35247a90ad07de6fb0cac2292be4f952337f98d2a2ac0f46b1aebbce033
Sep 04 23:41:03.049129 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:03.049094    2557 generic.go:356] "Generic (PLEG): container finished" podID="86ab9f3c-72be-4248-a3ba-f595a6a4723e" containerID="19a5e339ea3a761a22816ec688a33aa4be5a6f21c8f7aeed85dc054c04f205d5" exitCode=0
Sep 04 23:41:03.049613 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:03.049135    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx" event={"ID":"86ab9f3c-72be-4248-a3ba-f595a6a4723e","Type":"ContainerDied","Data":"19a5e339ea3a761a22816ec688a33aa4be5a6f21c8f7aeed85dc054c04f205d5"}
Sep 04 23:41:03.050977 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:03.050951    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-f6mmm" event={"ID":"836a759d-06ff-42c5-a5d3-70e88815ddd3","Type":"ContainerStarted","Data":"2dc37c2f4c52f4ed706531e048e54d9d4f957294fd91068ad4e030d29d76b128"}
Sep 04 23:41:03.051098 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:03.050987    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-f6mmm" event={"ID":"836a759d-06ff-42c5-a5d3-70e88815ddd3","Type":"ContainerStarted","Data":"9c70a35247a90ad07de6fb0cac2292be4f952337f98d2a2ac0f46b1aebbce033"}
Sep 04 23:41:03.090099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:03.090055    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-f6mmm" podStartSLOduration=2.090041842 podStartE2EDuration="2.090041842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:41:03.088399088 +0000 UTC m=+245.506217407" watchObservedRunningTime="2026-09-04 23:41:03.090041842 +0000 UTC m=+245.507860159"
Sep 04 23:41:04.057619 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:04.057587    2557 generic.go:356] "Generic (PLEG): container finished" podID="86ab9f3c-72be-4248-a3ba-f595a6a4723e" containerID="ce6021723f6ade12a3be392f8327090114289c9b08c4f3f4ae1eee14a93040bc" exitCode=0
Sep 04 23:41:04.057975 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:04.057669    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx" event={"ID":"86ab9f3c-72be-4248-a3ba-f595a6a4723e","Type":"ContainerDied","Data":"ce6021723f6ade12a3be392f8327090114289c9b08c4f3f4ae1eee14a93040bc"}
Sep 04 23:41:05.183197 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:05.183176    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"
Sep 04 23:41:05.273292 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:05.273268    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgpcx\" (UniqueName: \"kubernetes.io/projected/86ab9f3c-72be-4248-a3ba-f595a6a4723e-kube-api-access-jgpcx\") pod \"86ab9f3c-72be-4248-a3ba-f595a6a4723e\" (UID: \"86ab9f3c-72be-4248-a3ba-f595a6a4723e\") "
Sep 04 23:41:05.273411 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:05.273312    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86ab9f3c-72be-4248-a3ba-f595a6a4723e-bundle\") pod \"86ab9f3c-72be-4248-a3ba-f595a6a4723e\" (UID: \"86ab9f3c-72be-4248-a3ba-f595a6a4723e\") "
Sep 04 23:41:05.273411 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:05.273339    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86ab9f3c-72be-4248-a3ba-f595a6a4723e-util\") pod \"86ab9f3c-72be-4248-a3ba-f595a6a4723e\" (UID: \"86ab9f3c-72be-4248-a3ba-f595a6a4723e\") "
Sep 04 23:41:05.273682 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:05.273661    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ab9f3c-72be-4248-a3ba-f595a6a4723e-bundle" (OuterVolumeSpecName: "bundle") pod "86ab9f3c-72be-4248-a3ba-f595a6a4723e" (UID: "86ab9f3c-72be-4248-a3ba-f595a6a4723e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:41:05.275410 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:05.275379    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ab9f3c-72be-4248-a3ba-f595a6a4723e-kube-api-access-jgpcx" (OuterVolumeSpecName: "kube-api-access-jgpcx") pod "86ab9f3c-72be-4248-a3ba-f595a6a4723e" (UID: "86ab9f3c-72be-4248-a3ba-f595a6a4723e"). InnerVolumeSpecName "kube-api-access-jgpcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:41:05.278177 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:05.278150    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ab9f3c-72be-4248-a3ba-f595a6a4723e-util" (OuterVolumeSpecName: "util") pod "86ab9f3c-72be-4248-a3ba-f595a6a4723e" (UID: "86ab9f3c-72be-4248-a3ba-f595a6a4723e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:41:05.373845 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:05.373784    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgpcx\" (UniqueName: \"kubernetes.io/projected/86ab9f3c-72be-4248-a3ba-f595a6a4723e-kube-api-access-jgpcx\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:41:05.373845 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:05.373806    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86ab9f3c-72be-4248-a3ba-f595a6a4723e-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:41:05.373845 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:05.373816    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86ab9f3c-72be-4248-a3ba-f595a6a4723e-util\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:41:06.067003 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:06.066968    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx" event={"ID":"86ab9f3c-72be-4248-a3ba-f595a6a4723e","Type":"ContainerDied","Data":"412cc005f2f0abe044216953e7387cba01fb1994c0d71c481193fbfe9559d526"}
Sep 04 23:41:06.067003 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:06.067001    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhz7tx"
Sep 04 23:41:06.067200 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:06.067022    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="412cc005f2f0abe044216953e7387cba01fb1994c0d71c481193fbfe9559d526"
Sep 04 23:41:08.052686 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:08.052659    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5h4fw"
Sep 04 23:41:13.662220 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.662183    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bxs66"]
Sep 04 23:41:13.662675 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.662559    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ab9f3c-72be-4248-a3ba-f595a6a4723e" containerName="util"
Sep 04 23:41:13.662675 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.662575    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ab9f3c-72be-4248-a3ba-f595a6a4723e" containerName="util"
Sep 04 23:41:13.662675 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.662602    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ab9f3c-72be-4248-a3ba-f595a6a4723e" containerName="extract"
Sep 04 23:41:13.662675 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.662611    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ab9f3c-72be-4248-a3ba-f595a6a4723e" containerName="extract"
Sep 04 23:41:13.662675 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.662630    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ab9f3c-72be-4248-a3ba-f595a6a4723e" containerName="pull"
Sep 04 23:41:13.662675 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.662638    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ab9f3c-72be-4248-a3ba-f595a6a4723e" containerName="pull"
Sep 04 23:41:13.662982 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.662702    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="86ab9f3c-72be-4248-a3ba-f595a6a4723e" containerName="extract"
Sep 04 23:41:13.667404 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.667383    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bxs66"
Sep 04 23:41:13.669968 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.669930    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 04 23:41:13.670052 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.670004    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 04 23:41:13.671057 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.671037    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-cnw5t\""
Sep 04 23:41:13.678448 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.678424    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bxs66"]
Sep 04 23:41:13.725760 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.725737    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fc2d476-f685-46ba-9dc0-feb88ddea0e9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-bxs66\" (UID: \"1fc2d476-f685-46ba-9dc0-feb88ddea0e9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bxs66"
Sep 04 23:41:13.725853 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.725775    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhnjh\" (UniqueName: \"kubernetes.io/projected/1fc2d476-f685-46ba-9dc0-feb88ddea0e9-kube-api-access-vhnjh\") pod \"openshift-lws-operator-fd8ccff4c-bxs66\" (UID: \"1fc2d476-f685-46ba-9dc0-feb88ddea0e9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bxs66"
Sep 04 23:41:13.826040 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.826022    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhnjh\" (UniqueName: \"kubernetes.io/projected/1fc2d476-f685-46ba-9dc0-feb88ddea0e9-kube-api-access-vhnjh\") pod \"openshift-lws-operator-fd8ccff4c-bxs66\" (UID: \"1fc2d476-f685-46ba-9dc0-feb88ddea0e9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bxs66"
Sep 04 23:41:13.826137 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.826074    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fc2d476-f685-46ba-9dc0-feb88ddea0e9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-bxs66\" (UID: \"1fc2d476-f685-46ba-9dc0-feb88ddea0e9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bxs66"
Sep 04 23:41:13.826398 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.826384    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fc2d476-f685-46ba-9dc0-feb88ddea0e9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-bxs66\" (UID: \"1fc2d476-f685-46ba-9dc0-feb88ddea0e9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bxs66"
Sep 04 23:41:13.835314 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.835295    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhnjh\" (UniqueName: \"kubernetes.io/projected/1fc2d476-f685-46ba-9dc0-feb88ddea0e9-kube-api-access-vhnjh\") pod \"openshift-lws-operator-fd8ccff4c-bxs66\" (UID: \"1fc2d476-f685-46ba-9dc0-feb88ddea0e9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bxs66"
Sep 04 23:41:13.976366 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:13.976345    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bxs66"
Sep 04 23:41:14.264997 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:14.264920    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bxs66"]
Sep 04 23:41:14.268790 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:41:14.268765    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fc2d476_f685_46ba_9dc0_feb88ddea0e9.slice/crio-77be666e1b03b54defaaa088e8e26c168c42dbc2303b4a23646edd25b828daab WatchSource:0}: Error finding container 77be666e1b03b54defaaa088e8e26c168c42dbc2303b4a23646edd25b828daab: Status 404 returned error can't find the container with id 77be666e1b03b54defaaa088e8e26c168c42dbc2303b4a23646edd25b828daab
Sep 04 23:41:15.104629 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:15.104597    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bxs66" event={"ID":"1fc2d476-f685-46ba-9dc0-feb88ddea0e9","Type":"ContainerStarted","Data":"77be666e1b03b54defaaa088e8e26c168c42dbc2303b4a23646edd25b828daab"}
Sep 04 23:41:16.115709 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:16.115671    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bxs66" event={"ID":"1fc2d476-f685-46ba-9dc0-feb88ddea0e9","Type":"ContainerStarted","Data":"32021e72133bc507559464df1010a169ddbbaa41c5f3e3af5240e4f26d6ab127"}
Sep 04 23:41:16.131707 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:16.131663    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bxs66" podStartSLOduration=1.460988107 podStartE2EDuration="3.131648789s" totalImagesPullingTime="1.670660682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:41:16.130277659 +0000 UTC m=+258.548095977" watchObservedRunningTime="2026-09-04 23:41:16.131648789 +0000 UTC m=+258.549467098"
Sep 04 23:41:18.443799 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.443769    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"]
Sep 04 23:41:18.458873 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.458843    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"]
Sep 04 23:41:18.459047 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.458932    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"
Sep 04 23:41:18.461740 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.461718    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 04 23:41:18.461740 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.461733    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l949n\""
Sep 04 23:41:18.462909 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.462889    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 04 23:41:18.558209 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.558183    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-285tk\" (UniqueName: \"kubernetes.io/projected/c3eb517f-152d-4d19-bdce-1f911f3247b7-kube-api-access-285tk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q\" (UID: \"c3eb517f-152d-4d19-bdce-1f911f3247b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"
Sep 04 23:41:18.558321 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.558214    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3eb517f-152d-4d19-bdce-1f911f3247b7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q\" (UID: \"c3eb517f-152d-4d19-bdce-1f911f3247b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"
Sep 04 23:41:18.558321 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.558241    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3eb517f-152d-4d19-bdce-1f911f3247b7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q\" (UID: \"c3eb517f-152d-4d19-bdce-1f911f3247b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"
Sep 04 23:41:18.659366 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.659343    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3eb517f-152d-4d19-bdce-1f911f3247b7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q\" (UID: \"c3eb517f-152d-4d19-bdce-1f911f3247b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"
Sep 04 23:41:18.659474 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.659395    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-285tk\" (UniqueName: \"kubernetes.io/projected/c3eb517f-152d-4d19-bdce-1f911f3247b7-kube-api-access-285tk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q\" (UID: \"c3eb517f-152d-4d19-bdce-1f911f3247b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"
Sep 04 23:41:18.659474 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.659415    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3eb517f-152d-4d19-bdce-1f911f3247b7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q\" (UID: \"c3eb517f-152d-4d19-bdce-1f911f3247b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"
Sep 04 23:41:18.659738 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.659713    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3eb517f-152d-4d19-bdce-1f911f3247b7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q\" (UID: \"c3eb517f-152d-4d19-bdce-1f911f3247b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"
Sep 04 23:41:18.659802 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.659721    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3eb517f-152d-4d19-bdce-1f911f3247b7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q\" (UID: \"c3eb517f-152d-4d19-bdce-1f911f3247b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"
Sep 04 23:41:18.667126 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.667108    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-285tk\" (UniqueName: \"kubernetes.io/projected/c3eb517f-152d-4d19-bdce-1f911f3247b7-kube-api-access-285tk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q\" (UID: \"c3eb517f-152d-4d19-bdce-1f911f3247b7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"
Sep 04 23:41:18.767524 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:18.767506    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"
Sep 04 23:41:19.049520 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:19.049351    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"]
Sep 04 23:41:19.052527 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:41:19.052499    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3eb517f_152d_4d19_bdce_1f911f3247b7.slice/crio-c64fd29ab3c38902449c7ec654625102429564d6142e1d6424380933648a8227 WatchSource:0}: Error finding container c64fd29ab3c38902449c7ec654625102429564d6142e1d6424380933648a8227: Status 404 returned error can't find the container with id c64fd29ab3c38902449c7ec654625102429564d6142e1d6424380933648a8227
Sep 04 23:41:19.128895 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:19.128871    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q" event={"ID":"c3eb517f-152d-4d19-bdce-1f911f3247b7","Type":"ContainerStarted","Data":"c64fd29ab3c38902449c7ec654625102429564d6142e1d6424380933648a8227"}
Sep 04 23:41:20.134177 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:20.134145    2557 generic.go:356] "Generic (PLEG): container finished" podID="c3eb517f-152d-4d19-bdce-1f911f3247b7" containerID="ff006b6076f80998e13d97447052b8879414aa4b2b00606587722969295be8d7" exitCode=0
Sep 04 23:41:20.134562 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:20.134235    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q" event={"ID":"c3eb517f-152d-4d19-bdce-1f911f3247b7","Type":"ContainerDied","Data":"ff006b6076f80998e13d97447052b8879414aa4b2b00606587722969295be8d7"}
Sep 04 23:41:21.139870 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:21.139785    2557 generic.go:356] "Generic (PLEG): container finished" podID="c3eb517f-152d-4d19-bdce-1f911f3247b7" containerID="25059c20e2a7f020fa00c41fd745ebf151eb6a4eae6aa2ade6580a8500a278d9" exitCode=0
Sep 04 23:41:21.140254 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:21.139871    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q" event={"ID":"c3eb517f-152d-4d19-bdce-1f911f3247b7","Type":"ContainerDied","Data":"25059c20e2a7f020fa00c41fd745ebf151eb6a4eae6aa2ade6580a8500a278d9"}
Sep 04 23:41:22.145470 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:22.145431    2557 generic.go:356] "Generic (PLEG): container finished" podID="c3eb517f-152d-4d19-bdce-1f911f3247b7" containerID="ae1c38e68c9f579bd1809ae523567b5b2b5425244a3284d30e7d250908d95924" exitCode=0
Sep 04 23:41:22.145804 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:22.145516    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q" event={"ID":"c3eb517f-152d-4d19-bdce-1f911f3247b7","Type":"ContainerDied","Data":"ae1c38e68c9f579bd1809ae523567b5b2b5425244a3284d30e7d250908d95924"}
Sep 04 23:41:23.266099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:23.266077    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"
Sep 04 23:41:23.295145 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:23.295120    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3eb517f-152d-4d19-bdce-1f911f3247b7-bundle\") pod \"c3eb517f-152d-4d19-bdce-1f911f3247b7\" (UID: \"c3eb517f-152d-4d19-bdce-1f911f3247b7\") "
Sep 04 23:41:23.295266 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:23.295207    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-285tk\" (UniqueName: \"kubernetes.io/projected/c3eb517f-152d-4d19-bdce-1f911f3247b7-kube-api-access-285tk\") pod \"c3eb517f-152d-4d19-bdce-1f911f3247b7\" (UID: \"c3eb517f-152d-4d19-bdce-1f911f3247b7\") "
Sep 04 23:41:23.295266 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:23.295248    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3eb517f-152d-4d19-bdce-1f911f3247b7-util\") pod \"c3eb517f-152d-4d19-bdce-1f911f3247b7\" (UID: \"c3eb517f-152d-4d19-bdce-1f911f3247b7\") "
Sep 04 23:41:23.295892 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:23.295869    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3eb517f-152d-4d19-bdce-1f911f3247b7-bundle" (OuterVolumeSpecName: "bundle") pod "c3eb517f-152d-4d19-bdce-1f911f3247b7" (UID: "c3eb517f-152d-4d19-bdce-1f911f3247b7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:41:23.297214 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:23.297192    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3eb517f-152d-4d19-bdce-1f911f3247b7-kube-api-access-285tk" (OuterVolumeSpecName: "kube-api-access-285tk") pod "c3eb517f-152d-4d19-bdce-1f911f3247b7" (UID: "c3eb517f-152d-4d19-bdce-1f911f3247b7"). InnerVolumeSpecName "kube-api-access-285tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:41:23.300407 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:23.300370    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3eb517f-152d-4d19-bdce-1f911f3247b7-util" (OuterVolumeSpecName: "util") pod "c3eb517f-152d-4d19-bdce-1f911f3247b7" (UID: "c3eb517f-152d-4d19-bdce-1f911f3247b7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:41:23.396285 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:23.396262    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3eb517f-152d-4d19-bdce-1f911f3247b7-util\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:41:23.396285 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:23.396282    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3eb517f-152d-4d19-bdce-1f911f3247b7-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:41:23.396437 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:23.396292    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-285tk\" (UniqueName: \"kubernetes.io/projected/c3eb517f-152d-4d19-bdce-1f911f3247b7-kube-api-access-285tk\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:41:24.155892 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:24.155861    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q" event={"ID":"c3eb517f-152d-4d19-bdce-1f911f3247b7","Type":"ContainerDied","Data":"c64fd29ab3c38902449c7ec654625102429564d6142e1d6424380933648a8227"}
Sep 04 23:41:24.155892 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:24.155875    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w75q"
Sep 04 23:41:24.156093 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:24.155908    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c64fd29ab3c38902449c7ec654625102429564d6142e1d6424380933648a8227"
Sep 04 23:41:35.270494 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.270454    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"]
Sep 04 23:41:35.270990 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.270912    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3eb517f-152d-4d19-bdce-1f911f3247b7" containerName="pull"
Sep 04 23:41:35.270990 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.270931    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3eb517f-152d-4d19-bdce-1f911f3247b7" containerName="pull"
Sep 04 23:41:35.270990 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.270960    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3eb517f-152d-4d19-bdce-1f911f3247b7" containerName="util"
Sep 04 23:41:35.270990 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.270969    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3eb517f-152d-4d19-bdce-1f911f3247b7" containerName="util"
Sep 04 23:41:35.270990 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.270978    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3eb517f-152d-4d19-bdce-1f911f3247b7" containerName="extract"
Sep 04 23:41:35.270990 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.270986    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3eb517f-152d-4d19-bdce-1f911f3247b7" containerName="extract"
Sep 04 23:41:35.271223 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.271094    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3eb517f-152d-4d19-bdce-1f911f3247b7" containerName="extract"
Sep 04 23:41:35.274157 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.274140    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"
Sep 04 23:41:35.276791 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.276767    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 04 23:41:35.276902 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.276883    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l949n\""
Sep 04 23:41:35.278114 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.278095    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 04 23:41:35.290013 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.289989    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"]
Sep 04 23:41:35.379390 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.379360    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng\" (UID: \"2b6742c7-45b3-4311-8dd2-6ec2e69e716c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"
Sep 04 23:41:35.379546 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.379420    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lnnj\" (UniqueName: \"kubernetes.io/projected/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-kube-api-access-9lnnj\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng\" (UID: \"2b6742c7-45b3-4311-8dd2-6ec2e69e716c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"
Sep 04 23:41:35.379546 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.379445    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng\" (UID: \"2b6742c7-45b3-4311-8dd2-6ec2e69e716c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"
Sep 04 23:41:35.440270 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.440242    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"]
Sep 04 23:41:35.442435 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.442420    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:35.445275 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.445254    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 04 23:41:35.445402 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.445343    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 04 23:41:35.445611 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.445595    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 04 23:41:35.445756 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.445742    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jbpgd\""
Sep 04 23:41:35.454498 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.454477    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"]
Sep 04 23:41:35.480114 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.480089    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/662aa4c8-4bd8-4817-82c9-fcc85b0d9833-cert\") pod \"lws-controller-manager-58b866555c-m5t4p\" (UID: \"662aa4c8-4bd8-4817-82c9-fcc85b0d9833\") " pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:35.480207 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.480118    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcsnt\" (UniqueName: \"kubernetes.io/projected/662aa4c8-4bd8-4817-82c9-fcc85b0d9833-kube-api-access-kcsnt\") pod \"lws-controller-manager-58b866555c-m5t4p\" (UID: \"662aa4c8-4bd8-4817-82c9-fcc85b0d9833\") " pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:35.480207 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.480139    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lnnj\" (UniqueName: \"kubernetes.io/projected/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-kube-api-access-9lnnj\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng\" (UID: \"2b6742c7-45b3-4311-8dd2-6ec2e69e716c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"
Sep 04 23:41:35.480207 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.480162    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/662aa4c8-4bd8-4817-82c9-fcc85b0d9833-manager-config\") pod \"lws-controller-manager-58b866555c-m5t4p\" (UID: \"662aa4c8-4bd8-4817-82c9-fcc85b0d9833\") " pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:35.480325 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.480220    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng\" (UID: \"2b6742c7-45b3-4311-8dd2-6ec2e69e716c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"
Sep 04 23:41:35.480383 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.480363    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng\" (UID: \"2b6742c7-45b3-4311-8dd2-6ec2e69e716c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"
Sep 04 23:41:35.480425 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.480408    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/662aa4c8-4bd8-4817-82c9-fcc85b0d9833-metrics-cert\") pod \"lws-controller-manager-58b866555c-m5t4p\" (UID: \"662aa4c8-4bd8-4817-82c9-fcc85b0d9833\") " pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:35.480585 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.480570    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng\" (UID: \"2b6742c7-45b3-4311-8dd2-6ec2e69e716c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"
Sep 04 23:41:35.480646 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.480632    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng\" (UID: \"2b6742c7-45b3-4311-8dd2-6ec2e69e716c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"
Sep 04 23:41:35.488620 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.488595    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lnnj\" (UniqueName: \"kubernetes.io/projected/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-kube-api-access-9lnnj\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng\" (UID: \"2b6742c7-45b3-4311-8dd2-6ec2e69e716c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"
Sep 04 23:41:35.581034 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.580971    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/662aa4c8-4bd8-4817-82c9-fcc85b0d9833-cert\") pod \"lws-controller-manager-58b866555c-m5t4p\" (UID: \"662aa4c8-4bd8-4817-82c9-fcc85b0d9833\") " pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:35.581034 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.581001    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcsnt\" (UniqueName: \"kubernetes.io/projected/662aa4c8-4bd8-4817-82c9-fcc85b0d9833-kube-api-access-kcsnt\") pod \"lws-controller-manager-58b866555c-m5t4p\" (UID: \"662aa4c8-4bd8-4817-82c9-fcc85b0d9833\") " pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:35.581034 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.581024    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/662aa4c8-4bd8-4817-82c9-fcc85b0d9833-manager-config\") pod \"lws-controller-manager-58b866555c-m5t4p\" (UID: \"662aa4c8-4bd8-4817-82c9-fcc85b0d9833\") " pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:35.581212 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.581061    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/662aa4c8-4bd8-4817-82c9-fcc85b0d9833-metrics-cert\") pod \"lws-controller-manager-58b866555c-m5t4p\" (UID: \"662aa4c8-4bd8-4817-82c9-fcc85b0d9833\") " pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:35.581744 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.581721    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/662aa4c8-4bd8-4817-82c9-fcc85b0d9833-manager-config\") pod \"lws-controller-manager-58b866555c-m5t4p\" (UID: \"662aa4c8-4bd8-4817-82c9-fcc85b0d9833\") " pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:35.583456 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.583428    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/662aa4c8-4bd8-4817-82c9-fcc85b0d9833-cert\") pod \"lws-controller-manager-58b866555c-m5t4p\" (UID: \"662aa4c8-4bd8-4817-82c9-fcc85b0d9833\") " pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:35.583553 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.583461    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/662aa4c8-4bd8-4817-82c9-fcc85b0d9833-metrics-cert\") pod \"lws-controller-manager-58b866555c-m5t4p\" (UID: \"662aa4c8-4bd8-4817-82c9-fcc85b0d9833\") " pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:35.583553 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.583523    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"
Sep 04 23:41:35.591126 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.591105    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcsnt\" (UniqueName: \"kubernetes.io/projected/662aa4c8-4bd8-4817-82c9-fcc85b0d9833-kube-api-access-kcsnt\") pod \"lws-controller-manager-58b866555c-m5t4p\" (UID: \"662aa4c8-4bd8-4817-82c9-fcc85b0d9833\") " pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:35.752458 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.752427    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:35.858509 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:35.858466    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"]
Sep 04 23:41:35.860770 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:41:35.860740    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6742c7_45b3_4311_8dd2_6ec2e69e716c.slice/crio-bfedac2e938d02e59180204ebb56a87bed0b744d98084ec30ea90b9013b42ef3 WatchSource:0}: Error finding container bfedac2e938d02e59180204ebb56a87bed0b744d98084ec30ea90b9013b42ef3: Status 404 returned error can't find the container with id bfedac2e938d02e59180204ebb56a87bed0b744d98084ec30ea90b9013b42ef3
Sep 04 23:41:36.034048 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.034022    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"]
Sep 04 23:41:36.054808 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:41:36.054784    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662aa4c8_4bd8_4817_82c9_fcc85b0d9833.slice/crio-1cc01150d88e0da759059031b88d7c246ee8562ec4b09c16b869bc2b1d7b42f2 WatchSource:0}: Error finding container 1cc01150d88e0da759059031b88d7c246ee8562ec4b09c16b869bc2b1d7b42f2: Status 404 returned error can't find the container with id 1cc01150d88e0da759059031b88d7c246ee8562ec4b09c16b869bc2b1d7b42f2
Sep 04 23:41:36.206652 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.206617    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p" event={"ID":"662aa4c8-4bd8-4817-82c9-fcc85b0d9833","Type":"ContainerStarted","Data":"1cc01150d88e0da759059031b88d7c246ee8562ec4b09c16b869bc2b1d7b42f2"}
Sep 04 23:41:36.208180 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.208149    2557 generic.go:356] "Generic (PLEG): container finished" podID="2b6742c7-45b3-4311-8dd2-6ec2e69e716c" containerID="0f17d1aa45937c28afae3a0971e0e2b280ecd3b24d6a6fd8a5d853ee80d07909" exitCode=0
Sep 04 23:41:36.208286 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.208234    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng" event={"ID":"2b6742c7-45b3-4311-8dd2-6ec2e69e716c","Type":"ContainerDied","Data":"0f17d1aa45937c28afae3a0971e0e2b280ecd3b24d6a6fd8a5d853ee80d07909"}
Sep 04 23:41:36.208348 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.208283    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng" event={"ID":"2b6742c7-45b3-4311-8dd2-6ec2e69e716c","Type":"ContainerStarted","Data":"bfedac2e938d02e59180204ebb56a87bed0b744d98084ec30ea90b9013b42ef3"}
Sep 04 23:41:36.470240 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.470163    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"]
Sep 04 23:41:36.503333 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.503310    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"]
Sep 04 23:41:36.503468 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.503400    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.506270 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.506130    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mm9nl\""
Sep 04 23:41:36.506270 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.506140    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 04 23:41:36.506270 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.506177    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 04 23:41:36.506270 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.506217    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 04 23:41:36.506570 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.506394    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 04 23:41:36.588176 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.588153    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3068ccbb-0fca-4923-b3cb-e5731bfe1dfa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b9496bc86-w7fwm\" (UID: \"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.588275 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.588182    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3068ccbb-0fca-4923-b3cb-e5731bfe1dfa-tmp\") pod \"opendatahub-operator-controller-manager-6b9496bc86-w7fwm\" (UID: \"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.588275 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.588215    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3068ccbb-0fca-4923-b3cb-e5731bfe1dfa-manifests\") pod \"opendatahub-operator-controller-manager-6b9496bc86-w7fwm\" (UID: \"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.588351 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.588282    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwrv7\" (UniqueName: \"kubernetes.io/projected/3068ccbb-0fca-4923-b3cb-e5731bfe1dfa-kube-api-access-bwrv7\") pod \"opendatahub-operator-controller-manager-6b9496bc86-w7fwm\" (UID: \"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.588351 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.588327    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3068ccbb-0fca-4923-b3cb-e5731bfe1dfa-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b9496bc86-w7fwm\" (UID: \"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.688957 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.688909    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrv7\" (UniqueName: \"kubernetes.io/projected/3068ccbb-0fca-4923-b3cb-e5731bfe1dfa-kube-api-access-bwrv7\") pod \"opendatahub-operator-controller-manager-6b9496bc86-w7fwm\" (UID: \"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.689073 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.688977    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3068ccbb-0fca-4923-b3cb-e5731bfe1dfa-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b9496bc86-w7fwm\" (UID: \"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.689073 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.689010    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3068ccbb-0fca-4923-b3cb-e5731bfe1dfa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b9496bc86-w7fwm\" (UID: \"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.689073 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.689042    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3068ccbb-0fca-4923-b3cb-e5731bfe1dfa-tmp\") pod \"opendatahub-operator-controller-manager-6b9496bc86-w7fwm\" (UID: \"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.689228 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.689102    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3068ccbb-0fca-4923-b3cb-e5731bfe1dfa-manifests\") pod \"opendatahub-operator-controller-manager-6b9496bc86-w7fwm\" (UID: \"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.689440 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.689419    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3068ccbb-0fca-4923-b3cb-e5731bfe1dfa-manifests\") pod \"opendatahub-operator-controller-manager-6b9496bc86-w7fwm\" (UID: \"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.691075 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.691054    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3068ccbb-0fca-4923-b3cb-e5731bfe1dfa-tmp\") pod \"opendatahub-operator-controller-manager-6b9496bc86-w7fwm\" (UID: \"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.691383 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.691361    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3068ccbb-0fca-4923-b3cb-e5731bfe1dfa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b9496bc86-w7fwm\" (UID: \"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.691471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.691454    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3068ccbb-0fca-4923-b3cb-e5731bfe1dfa-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b9496bc86-w7fwm\" (UID: \"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.698463 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.698443    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwrv7\" (UniqueName: \"kubernetes.io/projected/3068ccbb-0fca-4923-b3cb-e5731bfe1dfa-kube-api-access-bwrv7\") pod \"opendatahub-operator-controller-manager-6b9496bc86-w7fwm\" (UID: \"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa\") " pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:36.815681 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:36.815612    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:37.113527 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:37.113498    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"]
Sep 04 23:41:37.115801 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:41:37.115771    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3068ccbb_0fca_4923_b3cb_e5731bfe1dfa.slice/crio-a9ee4151d519afa0135f3fce83745e97649fd296b5df5fd50b92a5a1f11eb5c6 WatchSource:0}: Error finding container a9ee4151d519afa0135f3fce83745e97649fd296b5df5fd50b92a5a1f11eb5c6: Status 404 returned error can't find the container with id a9ee4151d519afa0135f3fce83745e97649fd296b5df5fd50b92a5a1f11eb5c6
Sep 04 23:41:37.214726 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:37.214686    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm" event={"ID":"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa","Type":"ContainerStarted","Data":"a9ee4151d519afa0135f3fce83745e97649fd296b5df5fd50b92a5a1f11eb5c6"}
Sep 04 23:41:37.216751 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:37.216721    2557 generic.go:356] "Generic (PLEG): container finished" podID="2b6742c7-45b3-4311-8dd2-6ec2e69e716c" containerID="dcfbcc1b98f1e50f76d5b54af6e308cf929ef5c7a0cf03e9c0c34df376cf5190" exitCode=0
Sep 04 23:41:37.216868 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:37.216819    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng" event={"ID":"2b6742c7-45b3-4311-8dd2-6ec2e69e716c","Type":"ContainerDied","Data":"dcfbcc1b98f1e50f76d5b54af6e308cf929ef5c7a0cf03e9c0c34df376cf5190"}
Sep 04 23:41:38.225017 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:38.224517    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p" event={"ID":"662aa4c8-4bd8-4817-82c9-fcc85b0d9833","Type":"ContainerStarted","Data":"1847217605ec80aa3d7215fa7e87a2bf1eeb53a3d6b5448ee794d9cbf7ebcf79"}
Sep 04 23:41:38.225017 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:38.224637    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:38.227257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:38.227213    2557 generic.go:356] "Generic (PLEG): container finished" podID="2b6742c7-45b3-4311-8dd2-6ec2e69e716c" containerID="96c55599dc688dea1397d5d533adbcb2793f53eed9c065da2730a8b45a3d93b9" exitCode=0
Sep 04 23:41:38.227405 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:38.227299    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng" event={"ID":"2b6742c7-45b3-4311-8dd2-6ec2e69e716c","Type":"ContainerDied","Data":"96c55599dc688dea1397d5d533adbcb2793f53eed9c065da2730a8b45a3d93b9"}
Sep 04 23:41:38.243284 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:38.243227    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p" podStartSLOduration=1.683231521 podStartE2EDuration="3.243207553s" totalImagesPullingTime="1.559976032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:41:38.240286043 +0000 UTC m=+280.658104387" watchObservedRunningTime="2026-09-04 23:41:38.243207553 +0000 UTC m=+280.661025872"
Sep 04 23:41:40.073310 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:40.073283    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"
Sep 04 23:41:40.118015 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:40.117991    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-bundle\") pod \"2b6742c7-45b3-4311-8dd2-6ec2e69e716c\" (UID: \"2b6742c7-45b3-4311-8dd2-6ec2e69e716c\") "
Sep 04 23:41:40.118112 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:40.118025    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-util\") pod \"2b6742c7-45b3-4311-8dd2-6ec2e69e716c\" (UID: \"2b6742c7-45b3-4311-8dd2-6ec2e69e716c\") "
Sep 04 23:41:40.118112 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:40.118063    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lnnj\" (UniqueName: \"kubernetes.io/projected/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-kube-api-access-9lnnj\") pod \"2b6742c7-45b3-4311-8dd2-6ec2e69e716c\" (UID: \"2b6742c7-45b3-4311-8dd2-6ec2e69e716c\") "
Sep 04 23:41:40.118725 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:40.118700    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-bundle" (OuterVolumeSpecName: "bundle") pod "2b6742c7-45b3-4311-8dd2-6ec2e69e716c" (UID: "2b6742c7-45b3-4311-8dd2-6ec2e69e716c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:41:40.120214 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:40.120179    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-kube-api-access-9lnnj" (OuterVolumeSpecName: "kube-api-access-9lnnj") pod "2b6742c7-45b3-4311-8dd2-6ec2e69e716c" (UID: "2b6742c7-45b3-4311-8dd2-6ec2e69e716c"). InnerVolumeSpecName "kube-api-access-9lnnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:41:40.123737 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:40.123715    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-util" (OuterVolumeSpecName: "util") pod "2b6742c7-45b3-4311-8dd2-6ec2e69e716c" (UID: "2b6742c7-45b3-4311-8dd2-6ec2e69e716c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:41:40.219355 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:40.219328    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:41:40.219355 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:40.219352    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-util\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:41:40.219485 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:40.219361    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lnnj\" (UniqueName: \"kubernetes.io/projected/2b6742c7-45b3-4311-8dd2-6ec2e69e716c-kube-api-access-9lnnj\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:41:40.238355 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:40.238326    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm" event={"ID":"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa","Type":"ContainerStarted","Data":"b2a84621bbd14dc6c45ecad31da5539e8885705b0b5682ae92a3f3afbd087066"}
Sep 04 23:41:40.240355 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:40.240327    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng"
Sep 04 23:41:40.240448 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:40.240337    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07vbng" event={"ID":"2b6742c7-45b3-4311-8dd2-6ec2e69e716c","Type":"ContainerDied","Data":"bfedac2e938d02e59180204ebb56a87bed0b744d98084ec30ea90b9013b42ef3"}
Sep 04 23:41:40.240494 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:40.240451    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfedac2e938d02e59180204ebb56a87bed0b744d98084ec30ea90b9013b42ef3"
Sep 04 23:41:41.246830 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:41.246798    2557 generic.go:356] "Generic (PLEG): container finished" podID="3068ccbb-0fca-4923-b3cb-e5731bfe1dfa" containerID="b2a84621bbd14dc6c45ecad31da5539e8885705b0b5682ae92a3f3afbd087066" exitCode=0
Sep 04 23:41:41.247166 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:41.246882    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm" event={"ID":"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa","Type":"ContainerDied","Data":"b2a84621bbd14dc6c45ecad31da5539e8885705b0b5682ae92a3f3afbd087066"}
Sep 04 23:41:42.253388 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:42.253355    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm" event={"ID":"3068ccbb-0fca-4923-b3cb-e5731bfe1dfa","Type":"ContainerStarted","Data":"43b902b32db387ba62168c543281054c0a431b469d41d942a248d73e8ecc11a3"}
Sep 04 23:41:42.253772 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:42.253480    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:42.275631 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:42.275586    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm" podStartSLOduration=1.895089874 podStartE2EDuration="6.275572327s" totalImagesPullingTime="3.367378111s" totalInitContainerRuntime="1.013104342s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:41:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:41:42.273348667 +0000 UTC m=+284.691166976" watchObservedRunningTime="2026-09-04 23:41:42.275572327 +0000 UTC m=+284.693390644"
Sep 04 23:41:49.234602 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:49.234574    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58b866555c-m5t4p"
Sep 04 23:41:53.260284 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:53.260256    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b9496bc86-w7fwm"
Sep 04 23:41:57.972594 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:57.972567    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 04 23:41:57.973264 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:57.973248    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 04 23:41:57.975921 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:41:57.975906    2557 kubelet.go:1648] "Image garbage collection succeeded"
Sep 04 23:42:02.657911 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.657874    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"]
Sep 04 23:42:02.660257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.658310    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6742c7-45b3-4311-8dd2-6ec2e69e716c" containerName="util"
Sep 04 23:42:02.660257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.658325    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6742c7-45b3-4311-8dd2-6ec2e69e716c" containerName="util"
Sep 04 23:42:02.660257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.658333    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6742c7-45b3-4311-8dd2-6ec2e69e716c" containerName="extract"
Sep 04 23:42:02.660257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.658338    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6742c7-45b3-4311-8dd2-6ec2e69e716c" containerName="extract"
Sep 04 23:42:02.660257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.658352    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6742c7-45b3-4311-8dd2-6ec2e69e716c" containerName="pull"
Sep 04 23:42:02.660257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.658357    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6742c7-45b3-4311-8dd2-6ec2e69e716c" containerName="pull"
Sep 04 23:42:02.660257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.658409    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6742c7-45b3-4311-8dd2-6ec2e69e716c" containerName="extract"
Sep 04 23:42:02.661108 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.661094    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.663665 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.663639    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-bk968\""
Sep 04 23:42:02.663808 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.663672    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 04 23:42:02.663808 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.663682    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 04 23:42:02.663808 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.663782    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 04 23:42:02.672167 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.672145    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"]
Sep 04 23:42:02.788825 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.788802    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.788923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.788836    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ptv4\" (UniqueName: \"kubernetes.io/projected/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-kube-api-access-2ptv4\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.788923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.788858    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.788923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.788909    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.789049 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.788965    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.789049 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.788997    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.789049 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.789015    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.789049 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.789042    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.789172 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.789081    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.890248 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.890221    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.890356 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.890255    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.890356 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.890273    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.890356 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.890292    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.890356 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.890314    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.890571 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.890451    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.890571 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.890507    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ptv4\" (UniqueName: \"kubernetes.io/projected/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-kube-api-access-2ptv4\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.890571 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.890545    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.890731 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.890571    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.890731 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.890647    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.890731 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.890665    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.890905 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.890784    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.890905 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.890883    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.891470 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.891441    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.892640 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.892618    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.893036 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.893017    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.899015 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.898990    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.899511 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.899491    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ptv4\" (UniqueName: \"kubernetes.io/projected/9507b3c4-5133-4c67-ae0e-b6f7ce22d313-kube-api-access-2ptv4\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z\" (UID: \"9507b3c4-5133-4c67-ae0e-b6f7ce22d313\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:02.971385 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:02.971366    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:03.249450 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:03.248203    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"]
Sep 04 23:42:03.256364 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:03.255761    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 23:42:03.348674 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:03.348644    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z" event={"ID":"9507b3c4-5133-4c67-ae0e-b6f7ce22d313","Type":"ContainerStarted","Data":"0fa51a725570d343bb44efe67183d589a2aea50a44e6abcccaa7c968cd45f0bb"}
Sep 04 23:42:06.763293 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:06.763250    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:42:06.763533 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:06.763335    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:42:06.763533 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:06.763366    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:42:07.192675 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.192583    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4svvv"]
Sep 04 23:42:07.196399 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.196380    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4svvv"
Sep 04 23:42:07.199046 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.199024    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 04 23:42:07.199216 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.199058    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qm4qf\""
Sep 04 23:42:07.203085 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.203059    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4svvv"]
Sep 04 23:42:07.325537 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.325509    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4njk\" (UniqueName: \"kubernetes.io/projected/38d54642-90cf-4657-8eda-2e493f2dd61c-kube-api-access-d4njk\") pod \"odh-model-controller-7998c67dc4-4svvv\" (UID: \"38d54642-90cf-4657-8eda-2e493f2dd61c\") " pod="opendatahub/odh-model-controller-7998c67dc4-4svvv"
Sep 04 23:42:07.325696 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.325550    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38d54642-90cf-4657-8eda-2e493f2dd61c-cert\") pod \"odh-model-controller-7998c67dc4-4svvv\" (UID: \"38d54642-90cf-4657-8eda-2e493f2dd61c\") " pod="opendatahub/odh-model-controller-7998c67dc4-4svvv"
Sep 04 23:42:07.370353 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.370321    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z" event={"ID":"9507b3c4-5133-4c67-ae0e-b6f7ce22d313","Type":"ContainerStarted","Data":"964eacb533e213b8bd8617b1c84a4b7fe1c962eb36c70b60c4f24716928f637c"}
Sep 04 23:42:07.389748 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.389697    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z" podStartSLOduration=1.882628271 podStartE2EDuration="5.389683079s" totalImagesPullingTime="3.507054808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:42:07.387327815 +0000 UTC m=+309.805146135" watchObservedRunningTime="2026-09-04 23:42:07.389683079 +0000 UTC m=+309.807501396"
Sep 04 23:42:07.426517 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.426489    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4njk\" (UniqueName: \"kubernetes.io/projected/38d54642-90cf-4657-8eda-2e493f2dd61c-kube-api-access-d4njk\") pod \"odh-model-controller-7998c67dc4-4svvv\" (UID: \"38d54642-90cf-4657-8eda-2e493f2dd61c\") " pod="opendatahub/odh-model-controller-7998c67dc4-4svvv"
Sep 04 23:42:07.426681 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.426529    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38d54642-90cf-4657-8eda-2e493f2dd61c-cert\") pod \"odh-model-controller-7998c67dc4-4svvv\" (UID: \"38d54642-90cf-4657-8eda-2e493f2dd61c\") " pod="opendatahub/odh-model-controller-7998c67dc4-4svvv"
Sep 04 23:42:07.426681 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:42:07.426630    2557 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 04 23:42:07.426681 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:42:07.426678    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d54642-90cf-4657-8eda-2e493f2dd61c-cert podName:38d54642-90cf-4657-8eda-2e493f2dd61c nodeName:}" failed. No retries permitted until 2026-09-04 23:42:07.926662144 +0000 UTC m=+310.344480441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/38d54642-90cf-4657-8eda-2e493f2dd61c-cert") pod "odh-model-controller-7998c67dc4-4svvv" (UID: "38d54642-90cf-4657-8eda-2e493f2dd61c") : secret "odh-model-controller-webhook-cert" not found
Sep 04 23:42:07.435881 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.435854    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4njk\" (UniqueName: \"kubernetes.io/projected/38d54642-90cf-4657-8eda-2e493f2dd61c-kube-api-access-d4njk\") pod \"odh-model-controller-7998c67dc4-4svvv\" (UID: \"38d54642-90cf-4657-8eda-2e493f2dd61c\") " pod="opendatahub/odh-model-controller-7998c67dc4-4svvv"
Sep 04 23:42:07.762091 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.762063    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-w7662"]
Sep 04 23:42:07.765229 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.765210    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-w7662"
Sep 04 23:42:07.767539 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.767513    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 04 23:42:07.767648 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.767539    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-ztr7q\""
Sep 04 23:42:07.776298 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.776274    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-w7662"]
Sep 04 23:42:07.930885 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.930853    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98446e0d-67db-497e-83f2-ecbb86cb2f30-tls-certs\") pod \"model-serving-api-54d89bd5fc-w7662\" (UID: \"98446e0d-67db-497e-83f2-ecbb86cb2f30\") " pod="opendatahub/model-serving-api-54d89bd5fc-w7662"
Sep 04 23:42:07.931065 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.930897    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38d54642-90cf-4657-8eda-2e493f2dd61c-cert\") pod \"odh-model-controller-7998c67dc4-4svvv\" (UID: \"38d54642-90cf-4657-8eda-2e493f2dd61c\") " pod="opendatahub/odh-model-controller-7998c67dc4-4svvv"
Sep 04 23:42:07.931065 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.930992    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfsg7\" (UniqueName: \"kubernetes.io/projected/98446e0d-67db-497e-83f2-ecbb86cb2f30-kube-api-access-hfsg7\") pod \"model-serving-api-54d89bd5fc-w7662\" (UID: \"98446e0d-67db-497e-83f2-ecbb86cb2f30\") " pod="opendatahub/model-serving-api-54d89bd5fc-w7662"
Sep 04 23:42:07.933325 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.933293    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38d54642-90cf-4657-8eda-2e493f2dd61c-cert\") pod \"odh-model-controller-7998c67dc4-4svvv\" (UID: \"38d54642-90cf-4657-8eda-2e493f2dd61c\") " pod="opendatahub/odh-model-controller-7998c67dc4-4svvv"
Sep 04 23:42:07.972241 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.972216    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:07.976467 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:07.976448    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:08.032175 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:08.032081    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfsg7\" (UniqueName: \"kubernetes.io/projected/98446e0d-67db-497e-83f2-ecbb86cb2f30-kube-api-access-hfsg7\") pod \"model-serving-api-54d89bd5fc-w7662\" (UID: \"98446e0d-67db-497e-83f2-ecbb86cb2f30\") " pod="opendatahub/model-serving-api-54d89bd5fc-w7662"
Sep 04 23:42:08.032336 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:08.032191    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98446e0d-67db-497e-83f2-ecbb86cb2f30-tls-certs\") pod \"model-serving-api-54d89bd5fc-w7662\" (UID: \"98446e0d-67db-497e-83f2-ecbb86cb2f30\") " pod="opendatahub/model-serving-api-54d89bd5fc-w7662"
Sep 04 23:42:08.035231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:08.035204    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98446e0d-67db-497e-83f2-ecbb86cb2f30-tls-certs\") pod \"model-serving-api-54d89bd5fc-w7662\" (UID: \"98446e0d-67db-497e-83f2-ecbb86cb2f30\") " pod="opendatahub/model-serving-api-54d89bd5fc-w7662"
Sep 04 23:42:08.040549 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:08.040519    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfsg7\" (UniqueName: \"kubernetes.io/projected/98446e0d-67db-497e-83f2-ecbb86cb2f30-kube-api-access-hfsg7\") pod \"model-serving-api-54d89bd5fc-w7662\" (UID: \"98446e0d-67db-497e-83f2-ecbb86cb2f30\") " pod="opendatahub/model-serving-api-54d89bd5fc-w7662"
Sep 04 23:42:08.075488 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:08.075456    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-w7662"
Sep 04 23:42:08.109654 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:08.109623    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4svvv"
Sep 04 23:42:08.360550 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:08.360526    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-w7662"]
Sep 04 23:42:08.362169 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:42:08.362141    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98446e0d_67db_497e_83f2_ecbb86cb2f30.slice/crio-86eb995f7e079dce78322fe77d3e6699d47a722a1328d267a40776e85aa89274 WatchSource:0}: Error finding container 86eb995f7e079dce78322fe77d3e6699d47a722a1328d267a40776e85aa89274: Status 404 returned error can't find the container with id 86eb995f7e079dce78322fe77d3e6699d47a722a1328d267a40776e85aa89274
Sep 04 23:42:08.375687 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:08.375662    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-w7662" event={"ID":"98446e0d-67db-497e-83f2-ecbb86cb2f30","Type":"ContainerStarted","Data":"86eb995f7e079dce78322fe77d3e6699d47a722a1328d267a40776e85aa89274"}
Sep 04 23:42:08.375989 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:08.375969    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:08.376986 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:08.376968    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z"
Sep 04 23:42:08.394507 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:08.394488    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4svvv"]
Sep 04 23:42:08.397371 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:42:08.397327    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d54642_90cf_4657_8eda_2e493f2dd61c.slice/crio-4c360433ff7844a89757178a066b782ee79f5bde2857042ada30e926d55c370f WatchSource:0}: Error finding container 4c360433ff7844a89757178a066b782ee79f5bde2857042ada30e926d55c370f: Status 404 returned error can't find the container with id 4c360433ff7844a89757178a066b782ee79f5bde2857042ada30e926d55c370f
Sep 04 23:42:09.389568 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:09.389496    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4svvv" event={"ID":"38d54642-90cf-4657-8eda-2e493f2dd61c","Type":"ContainerStarted","Data":"4c360433ff7844a89757178a066b782ee79f5bde2857042ada30e926d55c370f"}
Sep 04 23:42:13.413552 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:13.413517    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-w7662" event={"ID":"98446e0d-67db-497e-83f2-ecbb86cb2f30","Type":"ContainerStarted","Data":"2e812b1c0a47253666d8575c18c91b8b419d530a94e1b4cdedc632378111ae86"}
Sep 04 23:42:13.414039 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:13.413588    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-w7662"
Sep 04 23:42:13.415115 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:13.415092    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4svvv" event={"ID":"38d54642-90cf-4657-8eda-2e493f2dd61c","Type":"ContainerStarted","Data":"54feaa5d752d376f39a5b3fed8908d1ec1b6856e57da8d872d95d2f08f6d229a"}
Sep 04 23:42:13.415257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:13.415150    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-4svvv"
Sep 04 23:42:13.430106 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:13.430060    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-w7662" podStartSLOduration=2.317172315 podStartE2EDuration="6.430047241s" totalImagesPullingTime="4.112874926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:42:13.429175566 +0000 UTC m=+315.846993885" watchObservedRunningTime="2026-09-04 23:42:13.430047241 +0000 UTC m=+315.847865559"
Sep 04 23:42:13.452282 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:13.452238    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-4svvv" podStartSLOduration=2.329732204 podStartE2EDuration="6.452224986s" totalImagesPullingTime="4.122492782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:42:13.451640789 +0000 UTC m=+315.869459117" watchObservedRunningTime="2026-09-04 23:42:13.452224986 +0000 UTC m=+315.870043305"
Sep 04 23:42:14.546486 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.546457    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"]
Sep 04 23:42:14.627380 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.627343    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"]
Sep 04 23:42:14.627547 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.627412    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.630400 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.630361    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 04 23:42:14.630400 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.630369    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-c8td6\""
Sep 04 23:42:14.630660 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.630630    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 04 23:42:14.630759 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.630698    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 04 23:42:14.794746 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.794710    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f272cabc-d99d-4b08-beea-4aa119a495c5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.794906 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.794775    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f272cabc-d99d-4b08-beea-4aa119a495c5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.794906 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.794839    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f272cabc-d99d-4b08-beea-4aa119a495c5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.794906 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.794859    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69rzc\" (UniqueName: \"kubernetes.io/projected/f272cabc-d99d-4b08-beea-4aa119a495c5-kube-api-access-69rzc\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.794906 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.794900    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f272cabc-d99d-4b08-beea-4aa119a495c5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.795091 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.794920    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f272cabc-d99d-4b08-beea-4aa119a495c5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.795091 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.794951    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f272cabc-d99d-4b08-beea-4aa119a495c5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.895784 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.895720    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f272cabc-d99d-4b08-beea-4aa119a495c5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.895784 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.895755    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f272cabc-d99d-4b08-beea-4aa119a495c5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.895784 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.895782    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f272cabc-d99d-4b08-beea-4aa119a495c5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.896029 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.895816    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f272cabc-d99d-4b08-beea-4aa119a495c5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.896029 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.895835    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f272cabc-d99d-4b08-beea-4aa119a495c5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.896029 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.895851    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69rzc\" (UniqueName: \"kubernetes.io/projected/f272cabc-d99d-4b08-beea-4aa119a495c5-kube-api-access-69rzc\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.896029 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.895875    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f272cabc-d99d-4b08-beea-4aa119a495c5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.896637 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.896601    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f272cabc-d99d-4b08-beea-4aa119a495c5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.898119 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.898096    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f272cabc-d99d-4b08-beea-4aa119a495c5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.898216 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.898163    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f272cabc-d99d-4b08-beea-4aa119a495c5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.898606 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.898583    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f272cabc-d99d-4b08-beea-4aa119a495c5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.898687 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.898586    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f272cabc-d99d-4b08-beea-4aa119a495c5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.903692 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.903668    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f272cabc-d99d-4b08-beea-4aa119a495c5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.904568 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.904547    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69rzc\" (UniqueName: \"kubernetes.io/projected/f272cabc-d99d-4b08-beea-4aa119a495c5-kube-api-access-69rzc\") pod \"istiod-openshift-gateway-5677ccb49b-j2lqk\" (UID: \"f272cabc-d99d-4b08-beea-4aa119a495c5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:14.938479 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:14.938460    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:15.245971 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:15.245932    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"]
Sep 04 23:42:15.248413 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:42:15.248385    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf272cabc_d99d_4b08_beea_4aa119a495c5.slice/crio-2e885f9af81aff886f4ccfcd002a15f8734ba888b338cb23cc210af56d1aa035 WatchSource:0}: Error finding container 2e885f9af81aff886f4ccfcd002a15f8734ba888b338cb23cc210af56d1aa035: Status 404 returned error can't find the container with id 2e885f9af81aff886f4ccfcd002a15f8734ba888b338cb23cc210af56d1aa035
Sep 04 23:42:15.426364 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:15.426338    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk" event={"ID":"f272cabc-d99d-4b08-beea-4aa119a495c5","Type":"ContainerStarted","Data":"2e885f9af81aff886f4ccfcd002a15f8734ba888b338cb23cc210af56d1aa035"}
Sep 04 23:42:18.263149 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:18.263114    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:42:18.263405 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:18.263177    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:42:18.442249 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:18.442213    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk" event={"ID":"f272cabc-d99d-4b08-beea-4aa119a495c5","Type":"ContainerStarted","Data":"5754322d00452881cff63fb556682f6fc0ed6ccfd9f281d7b058f99be01fc198"}
Sep 04 23:42:18.442390 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:18.442280    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:18.479659 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:18.479609    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk" podStartSLOduration=1.4677150270000001 podStartE2EDuration="4.47959465s" totalImagesPullingTime="3.011879623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:42:18.477115135 +0000 UTC m=+320.894933454" watchObservedRunningTime="2026-09-04 23:42:18.47959465 +0000 UTC m=+320.897412968"
Sep 04 23:42:19.449397 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:19.449364    2557 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-j2lqk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 04 23:42:19.450769 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:19.450741    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk" podUID="f272cabc-d99d-4b08-beea-4aa119a495c5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 23:42:22.447975 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:22.447926    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j2lqk"
Sep 04 23:42:24.422914 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:24.422886    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-4svvv"
Sep 04 23:42:24.424953 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:42:24.424922    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-w7662"
Sep 04 23:43:11.191482 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.191412    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"]
Sep 04 23:43:11.194683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.194667    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"
Sep 04 23:43:11.197183 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.197162    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 04 23:43:11.197183 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.197174    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l949n\""
Sep 04 23:43:11.198395 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.198374    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 04 23:43:11.202606 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.202583    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"]
Sep 04 23:43:11.291304 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.291271    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"]
Sep 04 23:43:11.294648 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.294629    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"
Sep 04 23:43:11.302622 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.302600    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"]
Sep 04 23:43:11.324759 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.324733    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg5tf\" (UniqueName: \"kubernetes.io/projected/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-kube-api-access-pg5tf\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj\" (UID: \"c73db03a-0efe-42f2-a49d-a5d6b1ddc486\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"
Sep 04 23:43:11.324850 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.324764    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj\" (UID: \"c73db03a-0efe-42f2-a49d-a5d6b1ddc486\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"
Sep 04 23:43:11.324908 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.324889    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj\" (UID: \"c73db03a-0efe-42f2-a49d-a5d6b1ddc486\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"
Sep 04 23:43:11.389348 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.389324    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"]
Sep 04 23:43:11.392825 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.392810    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"
Sep 04 23:43:11.400694 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.400672    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"]
Sep 04 23:43:11.425818 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.425793    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d352e43-fd9b-4bc4-90d0-91668e470878-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765\" (UID: \"4d352e43-fd9b-4bc4-90d0-91668e470878\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"
Sep 04 23:43:11.425927 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.425859    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d352e43-fd9b-4bc4-90d0-91668e470878-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765\" (UID: \"4d352e43-fd9b-4bc4-90d0-91668e470878\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"
Sep 04 23:43:11.426006 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.425964    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj\" (UID: \"c73db03a-0efe-42f2-a49d-a5d6b1ddc486\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"
Sep 04 23:43:11.426065 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.426020    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg5tf\" (UniqueName: \"kubernetes.io/projected/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-kube-api-access-pg5tf\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj\" (UID: \"c73db03a-0efe-42f2-a49d-a5d6b1ddc486\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"
Sep 04 23:43:11.426065 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.426045    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m9td\" (UniqueName: \"kubernetes.io/projected/4d352e43-fd9b-4bc4-90d0-91668e470878-kube-api-access-6m9td\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765\" (UID: \"4d352e43-fd9b-4bc4-90d0-91668e470878\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"
Sep 04 23:43:11.426171 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.426064    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj\" (UID: \"c73db03a-0efe-42f2-a49d-a5d6b1ddc486\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"
Sep 04 23:43:11.426342 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.426323    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj\" (UID: \"c73db03a-0efe-42f2-a49d-a5d6b1ddc486\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"
Sep 04 23:43:11.426402 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.426350    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj\" (UID: \"c73db03a-0efe-42f2-a49d-a5d6b1ddc486\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"
Sep 04 23:43:11.433524 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.433506    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg5tf\" (UniqueName: \"kubernetes.io/projected/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-kube-api-access-pg5tf\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj\" (UID: \"c73db03a-0efe-42f2-a49d-a5d6b1ddc486\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"
Sep 04 23:43:11.492192 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.492166    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"]
Sep 04 23:43:11.502207 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.502181    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"
Sep 04 23:43:11.503314 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.503290    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"]
Sep 04 23:43:11.504475 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.504458    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"
Sep 04 23:43:11.527985 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.527956    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m9td\" (UniqueName: \"kubernetes.io/projected/4d352e43-fd9b-4bc4-90d0-91668e470878-kube-api-access-6m9td\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765\" (UID: \"4d352e43-fd9b-4bc4-90d0-91668e470878\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"
Sep 04 23:43:11.528090 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.528014    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d352e43-fd9b-4bc4-90d0-91668e470878-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765\" (UID: \"4d352e43-fd9b-4bc4-90d0-91668e470878\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"
Sep 04 23:43:11.528090 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.528065    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq\" (UID: \"01e61f98-48ef-4ee0-b85b-ec4920b31fa3\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"
Sep 04 23:43:11.528207 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.528104    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d352e43-fd9b-4bc4-90d0-91668e470878-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765\" (UID: \"4d352e43-fd9b-4bc4-90d0-91668e470878\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"
Sep 04 23:43:11.528207 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.528142    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gzdr\" (UniqueName: \"kubernetes.io/projected/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-kube-api-access-7gzdr\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq\" (UID: \"01e61f98-48ef-4ee0-b85b-ec4920b31fa3\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"
Sep 04 23:43:11.528312 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.528206    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq\" (UID: \"01e61f98-48ef-4ee0-b85b-ec4920b31fa3\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"
Sep 04 23:43:11.528433 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.528415    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d352e43-fd9b-4bc4-90d0-91668e470878-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765\" (UID: \"4d352e43-fd9b-4bc4-90d0-91668e470878\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"
Sep 04 23:43:11.528496 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.528468    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d352e43-fd9b-4bc4-90d0-91668e470878-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765\" (UID: \"4d352e43-fd9b-4bc4-90d0-91668e470878\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"
Sep 04 23:43:11.535808 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.535782    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m9td\" (UniqueName: \"kubernetes.io/projected/4d352e43-fd9b-4bc4-90d0-91668e470878-kube-api-access-6m9td\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765\" (UID: \"4d352e43-fd9b-4bc4-90d0-91668e470878\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"
Sep 04 23:43:11.604344 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.604309    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"
Sep 04 23:43:11.628863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.628834    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwgb2\" (UniqueName: \"kubernetes.io/projected/0aa9528f-1f81-4a31-bd07-08e5ed879afc-kube-api-access-fwgb2\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds\" (UID: \"0aa9528f-1f81-4a31-bd07-08e5ed879afc\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"
Sep 04 23:43:11.629012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.628881    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq\" (UID: \"01e61f98-48ef-4ee0-b85b-ec4920b31fa3\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"
Sep 04 23:43:11.629012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.628949    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0aa9528f-1f81-4a31-bd07-08e5ed879afc-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds\" (UID: \"0aa9528f-1f81-4a31-bd07-08e5ed879afc\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"
Sep 04 23:43:11.629012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.629001    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gzdr\" (UniqueName: \"kubernetes.io/projected/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-kube-api-access-7gzdr\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq\" (UID: \"01e61f98-48ef-4ee0-b85b-ec4920b31fa3\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"
Sep 04 23:43:11.629164 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.629025    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0aa9528f-1f81-4a31-bd07-08e5ed879afc-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds\" (UID: \"0aa9528f-1f81-4a31-bd07-08e5ed879afc\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"
Sep 04 23:43:11.629164 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.629079    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq\" (UID: \"01e61f98-48ef-4ee0-b85b-ec4920b31fa3\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"
Sep 04 23:43:11.629261 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.629240    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq\" (UID: \"01e61f98-48ef-4ee0-b85b-ec4920b31fa3\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"
Sep 04 23:43:11.629417 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.629398    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq\" (UID: \"01e61f98-48ef-4ee0-b85b-ec4920b31fa3\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"
Sep 04 23:43:11.637340 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.637313    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gzdr\" (UniqueName: \"kubernetes.io/projected/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-kube-api-access-7gzdr\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq\" (UID: \"01e61f98-48ef-4ee0-b85b-ec4920b31fa3\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"
Sep 04 23:43:11.701984 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.701956    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"
Sep 04 23:43:11.730186 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.730157    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwgb2\" (UniqueName: \"kubernetes.io/projected/0aa9528f-1f81-4a31-bd07-08e5ed879afc-kube-api-access-fwgb2\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds\" (UID: \"0aa9528f-1f81-4a31-bd07-08e5ed879afc\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"
Sep 04 23:43:11.730326 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.730211    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0aa9528f-1f81-4a31-bd07-08e5ed879afc-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds\" (UID: \"0aa9528f-1f81-4a31-bd07-08e5ed879afc\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"
Sep 04 23:43:11.730326 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.730250    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0aa9528f-1f81-4a31-bd07-08e5ed879afc-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds\" (UID: \"0aa9528f-1f81-4a31-bd07-08e5ed879afc\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"
Sep 04 23:43:11.730651 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.730628    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0aa9528f-1f81-4a31-bd07-08e5ed879afc-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds\" (UID: \"0aa9528f-1f81-4a31-bd07-08e5ed879afc\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"
Sep 04 23:43:11.730728 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.730629    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0aa9528f-1f81-4a31-bd07-08e5ed879afc-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds\" (UID: \"0aa9528f-1f81-4a31-bd07-08e5ed879afc\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"
Sep 04 23:43:11.738394 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.738368    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwgb2\" (UniqueName: \"kubernetes.io/projected/0aa9528f-1f81-4a31-bd07-08e5ed879afc-kube-api-access-fwgb2\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds\" (UID: \"0aa9528f-1f81-4a31-bd07-08e5ed879afc\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"
Sep 04 23:43:11.777583 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.777562    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"]
Sep 04 23:43:11.779671 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:43:11.779642    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73db03a_0efe_42f2_a49d_a5d6b1ddc486.slice/crio-4f171721ea0400fb55a427eb0037647b74e74f9bbc75a5da9c0534c9516a3244 WatchSource:0}: Error finding container 4f171721ea0400fb55a427eb0037647b74e74f9bbc75a5da9c0534c9516a3244: Status 404 returned error can't find the container with id 4f171721ea0400fb55a427eb0037647b74e74f9bbc75a5da9c0534c9516a3244
Sep 04 23:43:11.812165 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.812142    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"
Sep 04 23:43:11.880526 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.880468    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"]
Sep 04 23:43:11.884024 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:43:11.883982    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d352e43_fd9b_4bc4_90d0_91668e470878.slice/crio-8d89944eb529f2dcb109df1b5213dac2f5a6d44bb47bdd2130f8613a9f0ad222 WatchSource:0}: Error finding container 8d89944eb529f2dcb109df1b5213dac2f5a6d44bb47bdd2130f8613a9f0ad222: Status 404 returned error can't find the container with id 8d89944eb529f2dcb109df1b5213dac2f5a6d44bb47bdd2130f8613a9f0ad222
Sep 04 23:43:11.973354 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:11.973332    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"]
Sep 04 23:43:11.974294 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:43:11.974272    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e61f98_48ef_4ee0_b85b_ec4920b31fa3.slice/crio-246c538d5663c4821e7b009886d187e71c10f4c1ed82db47796f70b35bfc2d63 WatchSource:0}: Error finding container 246c538d5663c4821e7b009886d187e71c10f4c1ed82db47796f70b35bfc2d63: Status 404 returned error can't find the container with id 246c538d5663c4821e7b009886d187e71c10f4c1ed82db47796f70b35bfc2d63
Sep 04 23:43:12.088878 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:12.088855    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"]
Sep 04 23:43:12.131625 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:43:12.131598    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa9528f_1f81_4a31_bd07_08e5ed879afc.slice/crio-8d78fde026ae5b20048bc94faf058cf578b3cb9bb856927f0e3b7aa1fc3505be WatchSource:0}: Error finding container 8d78fde026ae5b20048bc94faf058cf578b3cb9bb856927f0e3b7aa1fc3505be: Status 404 returned error can't find the container with id 8d78fde026ae5b20048bc94faf058cf578b3cb9bb856927f0e3b7aa1fc3505be
Sep 04 23:43:12.702177 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:12.702141    2557 generic.go:356] "Generic (PLEG): container finished" podID="0aa9528f-1f81-4a31-bd07-08e5ed879afc" containerID="f2e08403c8c40ed9faa2d948c2149c3c39137f5b323d5e4ded1182e7604eb94a" exitCode=0
Sep 04 23:43:12.702800 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:12.702231    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds" event={"ID":"0aa9528f-1f81-4a31-bd07-08e5ed879afc","Type":"ContainerDied","Data":"f2e08403c8c40ed9faa2d948c2149c3c39137f5b323d5e4ded1182e7604eb94a"}
Sep 04 23:43:12.702800 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:12.702288    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds" event={"ID":"0aa9528f-1f81-4a31-bd07-08e5ed879afc","Type":"ContainerStarted","Data":"8d78fde026ae5b20048bc94faf058cf578b3cb9bb856927f0e3b7aa1fc3505be"}
Sep 04 23:43:12.703785 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:12.703765    2557 generic.go:356] "Generic (PLEG): container finished" podID="4d352e43-fd9b-4bc4-90d0-91668e470878" containerID="c0e5d4f96cae5c321962c8659cd37a5bb5527195de48aafb7deb747b8340b89e" exitCode=0
Sep 04 23:43:12.703870 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:12.703847    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765" event={"ID":"4d352e43-fd9b-4bc4-90d0-91668e470878","Type":"ContainerDied","Data":"c0e5d4f96cae5c321962c8659cd37a5bb5527195de48aafb7deb747b8340b89e"}
Sep 04 23:43:12.703928 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:12.703896    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765" event={"ID":"4d352e43-fd9b-4bc4-90d0-91668e470878","Type":"ContainerStarted","Data":"8d89944eb529f2dcb109df1b5213dac2f5a6d44bb47bdd2130f8613a9f0ad222"}
Sep 04 23:43:12.705521 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:12.705502    2557 generic.go:356] "Generic (PLEG): container finished" podID="c73db03a-0efe-42f2-a49d-a5d6b1ddc486" containerID="9043bcc7729f12e5398bc32cb9744c51e9af87b06b25cfe6ebb008fabcff7bd5" exitCode=0
Sep 04 23:43:12.705603 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:12.705581    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj" event={"ID":"c73db03a-0efe-42f2-a49d-a5d6b1ddc486","Type":"ContainerDied","Data":"9043bcc7729f12e5398bc32cb9744c51e9af87b06b25cfe6ebb008fabcff7bd5"}
Sep 04 23:43:12.705659 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:12.705629    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj" event={"ID":"c73db03a-0efe-42f2-a49d-a5d6b1ddc486","Type":"ContainerStarted","Data":"4f171721ea0400fb55a427eb0037647b74e74f9bbc75a5da9c0534c9516a3244"}
Sep 04 23:43:12.706947 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:12.706918    2557 generic.go:356] "Generic (PLEG): container finished" podID="01e61f98-48ef-4ee0-b85b-ec4920b31fa3" containerID="a26f1d439e5738b550254c0330a3845fa92c7832b9789b93ccd081942af50271" exitCode=0
Sep 04 23:43:12.707004 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:12.706957    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq" event={"ID":"01e61f98-48ef-4ee0-b85b-ec4920b31fa3","Type":"ContainerDied","Data":"a26f1d439e5738b550254c0330a3845fa92c7832b9789b93ccd081942af50271"}
Sep 04 23:43:12.707004 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:12.706987    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq" event={"ID":"01e61f98-48ef-4ee0-b85b-ec4920b31fa3","Type":"ContainerStarted","Data":"246c538d5663c4821e7b009886d187e71c10f4c1ed82db47796f70b35bfc2d63"}
Sep 04 23:43:13.713532 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:13.713502    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq" event={"ID":"01e61f98-48ef-4ee0-b85b-ec4920b31fa3","Type":"ContainerStarted","Data":"5e13fc1fba36aa32e081b09cf2d97d8a00d0a2684a184214905e0dea4afa32be"}
Sep 04 23:43:13.715318 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:13.715294    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds" event={"ID":"0aa9528f-1f81-4a31-bd07-08e5ed879afc","Type":"ContainerStarted","Data":"920a9ee019faf005e4b82b3bc643720e3c19534dd5cb0410dde1cc4422a42ec0"}
Sep 04 23:43:13.717096 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:13.717076    2557 generic.go:356] "Generic (PLEG): container finished" podID="4d352e43-fd9b-4bc4-90d0-91668e470878" containerID="543ac1ff0d0d484d7ea5eb9ed0befe44d9b72f99f3d19b37a66944e55299d9a9" exitCode=0
Sep 04 23:43:13.717177 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:13.717114    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765" event={"ID":"4d352e43-fd9b-4bc4-90d0-91668e470878","Type":"ContainerDied","Data":"543ac1ff0d0d484d7ea5eb9ed0befe44d9b72f99f3d19b37a66944e55299d9a9"}
Sep 04 23:43:14.727499 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:14.727461    2557 generic.go:356] "Generic (PLEG): container finished" podID="01e61f98-48ef-4ee0-b85b-ec4920b31fa3" containerID="5e13fc1fba36aa32e081b09cf2d97d8a00d0a2684a184214905e0dea4afa32be" exitCode=0
Sep 04 23:43:14.727914 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:14.727547    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq" event={"ID":"01e61f98-48ef-4ee0-b85b-ec4920b31fa3","Type":"ContainerDied","Data":"5e13fc1fba36aa32e081b09cf2d97d8a00d0a2684a184214905e0dea4afa32be"}
Sep 04 23:43:14.729565 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:14.729540    2557 generic.go:356] "Generic (PLEG): container finished" podID="0aa9528f-1f81-4a31-bd07-08e5ed879afc" containerID="920a9ee019faf005e4b82b3bc643720e3c19534dd5cb0410dde1cc4422a42ec0" exitCode=0
Sep 04 23:43:14.729657 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:14.729571    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds" event={"ID":"0aa9528f-1f81-4a31-bd07-08e5ed879afc","Type":"ContainerDied","Data":"920a9ee019faf005e4b82b3bc643720e3c19534dd5cb0410dde1cc4422a42ec0"}
Sep 04 23:43:14.731826 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:14.731806    2557 generic.go:356] "Generic (PLEG): container finished" podID="4d352e43-fd9b-4bc4-90d0-91668e470878" containerID="d60103ca8ed7214fdb3f620f2198a27c8a35aaa09191e348508bdef6bd888bef" exitCode=0
Sep 04 23:43:14.731915 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:14.731830    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765" event={"ID":"4d352e43-fd9b-4bc4-90d0-91668e470878","Type":"ContainerDied","Data":"d60103ca8ed7214fdb3f620f2198a27c8a35aaa09191e348508bdef6bd888bef"}
Sep 04 23:43:14.733681 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:14.733664    2557 generic.go:356] "Generic (PLEG): container finished" podID="c73db03a-0efe-42f2-a49d-a5d6b1ddc486" containerID="4057c20c6d59042fb776f2123749b1cc0a3861dfef18ed35e2479bccc4d68164" exitCode=0
Sep 04 23:43:14.733817 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:14.733689    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj" event={"ID":"c73db03a-0efe-42f2-a49d-a5d6b1ddc486","Type":"ContainerDied","Data":"4057c20c6d59042fb776f2123749b1cc0a3861dfef18ed35e2479bccc4d68164"}
Sep 04 23:43:15.741339 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:15.741306    2557 generic.go:356] "Generic (PLEG): container finished" podID="0aa9528f-1f81-4a31-bd07-08e5ed879afc" containerID="14f874b72dea50b5bddcea9e9b7cc07128eda6cea8dbfa5243c7ba3132331170" exitCode=0
Sep 04 23:43:15.741727 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:15.741374    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds" event={"ID":"0aa9528f-1f81-4a31-bd07-08e5ed879afc","Type":"ContainerDied","Data":"14f874b72dea50b5bddcea9e9b7cc07128eda6cea8dbfa5243c7ba3132331170"}
Sep 04 23:43:15.743343 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:15.743319    2557 generic.go:356] "Generic (PLEG): container finished" podID="c73db03a-0efe-42f2-a49d-a5d6b1ddc486" containerID="1f92fd428bf3d40a7993d0823921bef4b330985e25108d59b2e97078236ac08c" exitCode=0
Sep 04 23:43:15.743472 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:15.743388    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj" event={"ID":"c73db03a-0efe-42f2-a49d-a5d6b1ddc486","Type":"ContainerDied","Data":"1f92fd428bf3d40a7993d0823921bef4b330985e25108d59b2e97078236ac08c"}
Sep 04 23:43:15.745408 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:15.745386    2557 generic.go:356] "Generic (PLEG): container finished" podID="01e61f98-48ef-4ee0-b85b-ec4920b31fa3" containerID="e5c1db259dfed9cf36ad1395f3d69502f9ba70d3cde5ee59585141513186d489" exitCode=0
Sep 04 23:43:15.745487 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:15.745449    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq" event={"ID":"01e61f98-48ef-4ee0-b85b-ec4920b31fa3","Type":"ContainerDied","Data":"e5c1db259dfed9cf36ad1395f3d69502f9ba70d3cde5ee59585141513186d489"}
Sep 04 23:43:15.873468 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:15.873445    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"
Sep 04 23:43:15.965139 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:15.965115    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d352e43-fd9b-4bc4-90d0-91668e470878-bundle\") pod \"4d352e43-fd9b-4bc4-90d0-91668e470878\" (UID: \"4d352e43-fd9b-4bc4-90d0-91668e470878\") "
Sep 04 23:43:15.965139 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:15.965142    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d352e43-fd9b-4bc4-90d0-91668e470878-util\") pod \"4d352e43-fd9b-4bc4-90d0-91668e470878\" (UID: \"4d352e43-fd9b-4bc4-90d0-91668e470878\") "
Sep 04 23:43:15.965288 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:15.965171    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m9td\" (UniqueName: \"kubernetes.io/projected/4d352e43-fd9b-4bc4-90d0-91668e470878-kube-api-access-6m9td\") pod \"4d352e43-fd9b-4bc4-90d0-91668e470878\" (UID: \"4d352e43-fd9b-4bc4-90d0-91668e470878\") "
Sep 04 23:43:15.965643 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:15.965617    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d352e43-fd9b-4bc4-90d0-91668e470878-bundle" (OuterVolumeSpecName: "bundle") pod "4d352e43-fd9b-4bc4-90d0-91668e470878" (UID: "4d352e43-fd9b-4bc4-90d0-91668e470878"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:43:15.967175 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:15.967155    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d352e43-fd9b-4bc4-90d0-91668e470878-kube-api-access-6m9td" (OuterVolumeSpecName: "kube-api-access-6m9td") pod "4d352e43-fd9b-4bc4-90d0-91668e470878" (UID: "4d352e43-fd9b-4bc4-90d0-91668e470878"). InnerVolumeSpecName "kube-api-access-6m9td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:43:15.970547 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:15.970516    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d352e43-fd9b-4bc4-90d0-91668e470878-util" (OuterVolumeSpecName: "util") pod "4d352e43-fd9b-4bc4-90d0-91668e470878" (UID: "4d352e43-fd9b-4bc4-90d0-91668e470878"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:43:16.065910 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:16.065890    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d352e43-fd9b-4bc4-90d0-91668e470878-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:43:16.065910 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:16.065909    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d352e43-fd9b-4bc4-90d0-91668e470878-util\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:43:16.066036 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:16.065919    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m9td\" (UniqueName: \"kubernetes.io/projected/4d352e43-fd9b-4bc4-90d0-91668e470878-kube-api-access-6m9td\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:43:16.756819 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:16.756782    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765" event={"ID":"4d352e43-fd9b-4bc4-90d0-91668e470878","Type":"ContainerDied","Data":"8d89944eb529f2dcb109df1b5213dac2f5a6d44bb47bdd2130f8613a9f0ad222"}
Sep 04 23:43:16.757355 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:16.756854    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d89944eb529f2dcb109df1b5213dac2f5a6d44bb47bdd2130f8613a9f0ad222"
Sep 04 23:43:16.757355 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:16.756966    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c28765"
Sep 04 23:43:16.890490 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:16.890466    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"
Sep 04 23:43:16.946125 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:16.946102    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"
Sep 04 23:43:16.948431 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:16.948413    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"
Sep 04 23:43:16.972737 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:16.972715    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg5tf\" (UniqueName: \"kubernetes.io/projected/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-kube-api-access-pg5tf\") pod \"c73db03a-0efe-42f2-a49d-a5d6b1ddc486\" (UID: \"c73db03a-0efe-42f2-a49d-a5d6b1ddc486\") "
Sep 04 23:43:16.972854 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:16.972760    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-bundle\") pod \"c73db03a-0efe-42f2-a49d-a5d6b1ddc486\" (UID: \"c73db03a-0efe-42f2-a49d-a5d6b1ddc486\") "
Sep 04 23:43:16.972854 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:16.972794    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-util\") pod \"c73db03a-0efe-42f2-a49d-a5d6b1ddc486\" (UID: \"c73db03a-0efe-42f2-a49d-a5d6b1ddc486\") "
Sep 04 23:43:16.973186 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:16.973166    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-bundle" (OuterVolumeSpecName: "bundle") pod "c73db03a-0efe-42f2-a49d-a5d6b1ddc486" (UID: "c73db03a-0efe-42f2-a49d-a5d6b1ddc486"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:43:16.974707 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:16.974686    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-kube-api-access-pg5tf" (OuterVolumeSpecName: "kube-api-access-pg5tf") pod "c73db03a-0efe-42f2-a49d-a5d6b1ddc486" (UID: "c73db03a-0efe-42f2-a49d-a5d6b1ddc486"). InnerVolumeSpecName "kube-api-access-pg5tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:43:17.054874 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.054812    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-util" (OuterVolumeSpecName: "util") pod "c73db03a-0efe-42f2-a49d-a5d6b1ddc486" (UID: "c73db03a-0efe-42f2-a49d-a5d6b1ddc486"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:43:17.073968 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.073929    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gzdr\" (UniqueName: \"kubernetes.io/projected/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-kube-api-access-7gzdr\") pod \"01e61f98-48ef-4ee0-b85b-ec4920b31fa3\" (UID: \"01e61f98-48ef-4ee0-b85b-ec4920b31fa3\") "
Sep 04 23:43:17.074070 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.074001    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0aa9528f-1f81-4a31-bd07-08e5ed879afc-util\") pod \"0aa9528f-1f81-4a31-bd07-08e5ed879afc\" (UID: \"0aa9528f-1f81-4a31-bd07-08e5ed879afc\") "
Sep 04 23:43:17.074070 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.074050    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-util\") pod \"01e61f98-48ef-4ee0-b85b-ec4920b31fa3\" (UID: \"01e61f98-48ef-4ee0-b85b-ec4920b31fa3\") "
Sep 04 23:43:17.074159 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.074086    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-bundle\") pod \"01e61f98-48ef-4ee0-b85b-ec4920b31fa3\" (UID: \"01e61f98-48ef-4ee0-b85b-ec4920b31fa3\") "
Sep 04 23:43:17.074159 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.074122    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwgb2\" (UniqueName: \"kubernetes.io/projected/0aa9528f-1f81-4a31-bd07-08e5ed879afc-kube-api-access-fwgb2\") pod \"0aa9528f-1f81-4a31-bd07-08e5ed879afc\" (UID: \"0aa9528f-1f81-4a31-bd07-08e5ed879afc\") "
Sep 04 23:43:17.074159 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.074136    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0aa9528f-1f81-4a31-bd07-08e5ed879afc-bundle\") pod \"0aa9528f-1f81-4a31-bd07-08e5ed879afc\" (UID: \"0aa9528f-1f81-4a31-bd07-08e5ed879afc\") "
Sep 04 23:43:17.074403 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.074357    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-util\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:43:17.074403 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.074380    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg5tf\" (UniqueName: \"kubernetes.io/projected/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-kube-api-access-pg5tf\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:43:17.074403 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.074395    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c73db03a-0efe-42f2-a49d-a5d6b1ddc486-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:43:17.074662 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.074636    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-bundle" (OuterVolumeSpecName: "bundle") pod "01e61f98-48ef-4ee0-b85b-ec4920b31fa3" (UID: "01e61f98-48ef-4ee0-b85b-ec4920b31fa3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:43:17.074892 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.074871    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa9528f-1f81-4a31-bd07-08e5ed879afc-bundle" (OuterVolumeSpecName: "bundle") pod "0aa9528f-1f81-4a31-bd07-08e5ed879afc" (UID: "0aa9528f-1f81-4a31-bd07-08e5ed879afc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:43:17.076257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.076237    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-kube-api-access-7gzdr" (OuterVolumeSpecName: "kube-api-access-7gzdr") pod "01e61f98-48ef-4ee0-b85b-ec4920b31fa3" (UID: "01e61f98-48ef-4ee0-b85b-ec4920b31fa3"). InnerVolumeSpecName "kube-api-access-7gzdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:43:17.076519 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.076501    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa9528f-1f81-4a31-bd07-08e5ed879afc-kube-api-access-fwgb2" (OuterVolumeSpecName: "kube-api-access-fwgb2") pod "0aa9528f-1f81-4a31-bd07-08e5ed879afc" (UID: "0aa9528f-1f81-4a31-bd07-08e5ed879afc"). InnerVolumeSpecName "kube-api-access-fwgb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:43:17.080030 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.080001    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-util" (OuterVolumeSpecName: "util") pod "01e61f98-48ef-4ee0-b85b-ec4920b31fa3" (UID: "01e61f98-48ef-4ee0-b85b-ec4920b31fa3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:43:17.080610 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.080580    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa9528f-1f81-4a31-bd07-08e5ed879afc-util" (OuterVolumeSpecName: "util") pod "0aa9528f-1f81-4a31-bd07-08e5ed879afc" (UID: "0aa9528f-1f81-4a31-bd07-08e5ed879afc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:43:17.174699 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.174676    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gzdr\" (UniqueName: \"kubernetes.io/projected/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-kube-api-access-7gzdr\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:43:17.174699 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.174697    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0aa9528f-1f81-4a31-bd07-08e5ed879afc-util\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:43:17.174811 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.174706    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-util\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:43:17.174811 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.174714    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01e61f98-48ef-4ee0-b85b-ec4920b31fa3-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:43:17.174811 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.174721    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwgb2\" (UniqueName: \"kubernetes.io/projected/0aa9528f-1f81-4a31-bd07-08e5ed879afc-kube-api-access-fwgb2\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:43:17.174811 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.174730    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0aa9528f-1f81-4a31-bd07-08e5ed879afc-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:43:17.764034 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.764007    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds"
Sep 04 23:43:17.764034 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.764011    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539xtds" event={"ID":"0aa9528f-1f81-4a31-bd07-08e5ed879afc","Type":"ContainerDied","Data":"8d78fde026ae5b20048bc94faf058cf578b3cb9bb856927f0e3b7aa1fc3505be"}
Sep 04 23:43:17.764480 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.764052    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d78fde026ae5b20048bc94faf058cf578b3cb9bb856927f0e3b7aa1fc3505be"
Sep 04 23:43:17.765869 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.765849    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj" event={"ID":"c73db03a-0efe-42f2-a49d-a5d6b1ddc486","Type":"ContainerDied","Data":"4f171721ea0400fb55a427eb0037647b74e74f9bbc75a5da9c0534c9516a3244"}
Sep 04 23:43:17.766001 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.765882    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f171721ea0400fb55a427eb0037647b74e74f9bbc75a5da9c0534c9516a3244"
Sep 04 23:43:17.766001 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.765859    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0n28nj"
Sep 04 23:43:17.768141 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.768121    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq"
Sep 04 23:43:17.768229 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.768161    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpzfzq" event={"ID":"01e61f98-48ef-4ee0-b85b-ec4920b31fa3","Type":"ContainerDied","Data":"246c538d5663c4821e7b009886d187e71c10f4c1ed82db47796f70b35bfc2d63"}
Sep 04 23:43:17.768229 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:43:17.768196    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="246c538d5663c4821e7b009886d187e71c10f4c1ed82db47796f70b35bfc2d63"
Sep 04 23:44:20.037559 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037521    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037858    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d352e43-fd9b-4bc4-90d0-91668e470878" containerName="extract"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037869    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d352e43-fd9b-4bc4-90d0-91668e470878" containerName="extract"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037876    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e61f98-48ef-4ee0-b85b-ec4920b31fa3" containerName="extract"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037881    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e61f98-48ef-4ee0-b85b-ec4920b31fa3" containerName="extract"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037892    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73db03a-0efe-42f2-a49d-a5d6b1ddc486" containerName="util"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037898    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73db03a-0efe-42f2-a49d-a5d6b1ddc486" containerName="util"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037904    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e61f98-48ef-4ee0-b85b-ec4920b31fa3" containerName="pull"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037909    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e61f98-48ef-4ee0-b85b-ec4920b31fa3" containerName="pull"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037915    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d352e43-fd9b-4bc4-90d0-91668e470878" containerName="util"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037920    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d352e43-fd9b-4bc4-90d0-91668e470878" containerName="util"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037926    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73db03a-0efe-42f2-a49d-a5d6b1ddc486" containerName="extract"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037932    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73db03a-0efe-42f2-a49d-a5d6b1ddc486" containerName="extract"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037955    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aa9528f-1f81-4a31-bd07-08e5ed879afc" containerName="extract"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037960    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aa9528f-1f81-4a31-bd07-08e5ed879afc" containerName="extract"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037967    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73db03a-0efe-42f2-a49d-a5d6b1ddc486" containerName="pull"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037971    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73db03a-0efe-42f2-a49d-a5d6b1ddc486" containerName="pull"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037978    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e61f98-48ef-4ee0-b85b-ec4920b31fa3" containerName="util"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037983    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e61f98-48ef-4ee0-b85b-ec4920b31fa3" containerName="util"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037991    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aa9528f-1f81-4a31-bd07-08e5ed879afc" containerName="util"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.037996    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aa9528f-1f81-4a31-bd07-08e5ed879afc" containerName="util"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.038001    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aa9528f-1f81-4a31-bd07-08e5ed879afc" containerName="pull"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.038006    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aa9528f-1f81-4a31-bd07-08e5ed879afc" containerName="pull"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.038015    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d352e43-fd9b-4bc4-90d0-91668e470878" containerName="pull"
Sep 04 23:44:20.038050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.038020    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d352e43-fd9b-4bc4-90d0-91668e470878" containerName="pull"
Sep 04 23:44:20.038727 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.038072    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73db03a-0efe-42f2-a49d-a5d6b1ddc486" containerName="extract"
Sep 04 23:44:20.038727 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.038080    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e61f98-48ef-4ee0-b85b-ec4920b31fa3" containerName="extract"
Sep 04 23:44:20.038727 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.038088    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d352e43-fd9b-4bc4-90d0-91668e470878" containerName="extract"
Sep 04 23:44:20.038727 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.038094    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0aa9528f-1f81-4a31-bd07-08e5ed879afc" containerName="extract"
Sep 04 23:44:20.040810 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.040794    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"
Sep 04 23:44:20.043201 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.043166    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 04 23:44:20.043201 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.043166    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 04 23:44:20.043417 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.043289    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 04 23:44:20.044391 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.044375    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jgrf6\""
Sep 04 23:44:20.050728 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.050705    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:44:20.079039 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.079014    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:44:20.148738 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.148708    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fcr5\" (UniqueName: \"kubernetes.io/projected/fbc0ecf2-7717-4c7a-a285-20f4e70dce6f-kube-api-access-9fcr5\") pod \"limitador-limitador-85d9db4d5d-jcwjt\" (UID: \"fbc0ecf2-7717-4c7a-a285-20f4e70dce6f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"
Sep 04 23:44:20.148873 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.148750    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fbc0ecf2-7717-4c7a-a285-20f4e70dce6f-config-file\") pod \"limitador-limitador-85d9db4d5d-jcwjt\" (UID: \"fbc0ecf2-7717-4c7a-a285-20f4e70dce6f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"
Sep 04 23:44:20.249223 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.249194    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fbc0ecf2-7717-4c7a-a285-20f4e70dce6f-config-file\") pod \"limitador-limitador-85d9db4d5d-jcwjt\" (UID: \"fbc0ecf2-7717-4c7a-a285-20f4e70dce6f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"
Sep 04 23:44:20.249359 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.249314    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fcr5\" (UniqueName: \"kubernetes.io/projected/fbc0ecf2-7717-4c7a-a285-20f4e70dce6f-kube-api-access-9fcr5\") pod \"limitador-limitador-85d9db4d5d-jcwjt\" (UID: \"fbc0ecf2-7717-4c7a-a285-20f4e70dce6f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"
Sep 04 23:44:20.249788 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.249769    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fbc0ecf2-7717-4c7a-a285-20f4e70dce6f-config-file\") pod \"limitador-limitador-85d9db4d5d-jcwjt\" (UID: \"fbc0ecf2-7717-4c7a-a285-20f4e70dce6f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"
Sep 04 23:44:20.256728 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.256708    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fcr5\" (UniqueName: \"kubernetes.io/projected/fbc0ecf2-7717-4c7a-a285-20f4e70dce6f-kube-api-access-9fcr5\") pod \"limitador-limitador-85d9db4d5d-jcwjt\" (UID: \"fbc0ecf2-7717-4c7a-a285-20f4e70dce6f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"
Sep 04 23:44:20.351876 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.351773    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"
Sep 04 23:44:20.621581 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:20.621550    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:44:20.623238 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:44:20.623208    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc0ecf2_7717_4c7a_a285_20f4e70dce6f.slice/crio-cbe4c0b3d9655366eba394ae613e8f9f6a12d0c014d70b460769863680ca6e2e WatchSource:0}: Error finding container cbe4c0b3d9655366eba394ae613e8f9f6a12d0c014d70b460769863680ca6e2e: Status 404 returned error can't find the container with id cbe4c0b3d9655366eba394ae613e8f9f6a12d0c014d70b460769863680ca6e2e
Sep 04 23:44:21.084806 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:21.084768    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt" event={"ID":"fbc0ecf2-7717-4c7a-a285-20f4e70dce6f","Type":"ContainerStarted","Data":"cbe4c0b3d9655366eba394ae613e8f9f6a12d0c014d70b460769863680ca6e2e"}
Sep 04 23:44:23.095342 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:23.095263    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt" event={"ID":"fbc0ecf2-7717-4c7a-a285-20f4e70dce6f","Type":"ContainerStarted","Data":"595db3a84ab277343baa90888644b2d6edfc38163e25c029866fa0c6132454cb"}
Sep 04 23:44:23.095681 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:23.095367    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"
Sep 04 23:44:23.112978 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:23.112918    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt" podStartSLOduration=0.971118972 podStartE2EDuration="3.112904689s" totalImagesPullingTime="2.141785717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:44:23.111062593 +0000 UTC m=+445.528880917" watchObservedRunningTime="2026-09-04 23:44:23.112904689 +0000 UTC m=+445.530723010"
Sep 04 23:44:34.100204 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:44:34.100175    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"
Sep 04 23:45:34.564931 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:45:34.564899    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:45:54.610827 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:45:54.610743    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:45:59.917029 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:45:59.916998    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:46:15.716792 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:15.716755    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:46:23.511418 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:23.511385    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:46:38.917732 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:38.917696    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:46:45.216188 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:45.216151    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:46:52.310562 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.310506    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"]
Sep 04 23:46:52.314365 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.314343    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.318089 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.318063    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 04 23:46:52.318211 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.318144    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 04 23:46:52.318211 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.318063    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 04 23:46:52.318211 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.318076    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rfvrs\""
Sep 04 23:46:52.323718 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.323689    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"]
Sep 04 23:46:52.370832 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.370810    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.370961 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.370838    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.370961 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.370865    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.370961 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.370884    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.370961 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.370947    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.371106 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.371050    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpnvz\" (UniqueName: \"kubernetes.io/projected/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-kube-api-access-lpnvz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.472115 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.472089    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpnvz\" (UniqueName: \"kubernetes.io/projected/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-kube-api-access-lpnvz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.472331 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.472298    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.472478 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.472446    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.472587 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.472515    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.472587 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.472541    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.472587 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.472558    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.472740 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.472678    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.472824 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.472798    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.472924 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.472857    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.474827 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.474806    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.475087 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.475067    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.479312 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.479289    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpnvz\" (UniqueName: \"kubernetes.io/projected/58da7198-ec0f-4a2f-a77e-ee68c4c34cd2-kube-api-access-lpnvz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f\" (UID: \"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.624075 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.624011    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:46:52.895798 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:52.895775    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"]
Sep 04 23:46:52.897720 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:46:52.897693    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58da7198_ec0f_4a2f_a77e_ee68c4c34cd2.slice/crio-3b944e7b595a3b9819cdf9d8ca42eec1b72249d2a0708cd2be9e071aae09dbdd WatchSource:0}: Error finding container 3b944e7b595a3b9819cdf9d8ca42eec1b72249d2a0708cd2be9e071aae09dbdd: Status 404 returned error can't find the container with id 3b944e7b595a3b9819cdf9d8ca42eec1b72249d2a0708cd2be9e071aae09dbdd
Sep 04 23:46:53.838957 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:53.838812    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f" event={"ID":"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2","Type":"ContainerStarted","Data":"3b944e7b595a3b9819cdf9d8ca42eec1b72249d2a0708cd2be9e071aae09dbdd"}
Sep 04 23:46:54.816662 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:54.816632    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:46:54.845100 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:54.845073    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f" event={"ID":"58da7198-ec0f-4a2f-a77e-ee68c4c34cd2","Type":"ContainerStarted","Data":"d578954b32c71d9be75fcd676a3eeba90bab60418420e09590e696722115af93"}
Sep 04 23:46:54.861039 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:54.860994    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f" podStartSLOduration=1.920345244 podStartE2EDuration="2.860979546s" totalImagesPullingTime="940.634302ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:46:54.859802938 +0000 UTC m=+597.277621259" watchObservedRunningTime="2026-09-04 23:46:54.860979546 +0000 UTC m=+597.278797883"
Sep 04 23:46:58.008703 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:58.008670    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 04 23:46:58.009352 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:46:58.009323    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 04 23:47:02.624235 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:47:02.624200    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:47:02.624645 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:47:02.624246    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:47:02.636614 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:47:02.636594    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:47:02.896411 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:47:02.896349    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f"
Sep 04 23:48:17.260666 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.260625    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"]
Sep 04 23:48:17.264268 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.264251    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.266950 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.266913    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d8b443-openshift-default-dockercfg-qz624\""
Sep 04 23:48:17.276988 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.276963    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"]
Sep 04 23:48:17.403114 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.403083    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s57nh\" (UniqueName: \"kubernetes.io/projected/b149e027-57c4-47fd-a20b-af8c0969fd08-kube-api-access-s57nh\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.403114 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.403114    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-credential-socket\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.403329 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.403154    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-data\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.403329 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.403170    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b149e027-57c4-47fd-a20b-af8c0969fd08-istiod-ca-cert\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.403329 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.403239    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-token\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.403329 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.403276    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-podinfo\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.403329 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.403309    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-workload-certs\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.403329 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.403325    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-envoy\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.403537 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.403360    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-workload-socket\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.504144 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.504118    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-workload-socket\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.504275 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.504160    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s57nh\" (UniqueName: \"kubernetes.io/projected/b149e027-57c4-47fd-a20b-af8c0969fd08-kube-api-access-s57nh\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.504275 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.504178    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-credential-socket\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.504275 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.504214    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-data\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.504407 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.504322    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b149e027-57c4-47fd-a20b-af8c0969fd08-istiod-ca-cert\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.504407 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.504384    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-token\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.504520 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.504415    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-podinfo\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.504520 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.504481    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-workload-certs\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.504607 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.504546    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-credential-socket\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.505109 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.504506    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-envoy\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.505254 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.505128    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-workload-certs\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.505254 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.505180    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-data\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.508958 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.504547    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-workload-socket\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.508958 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.505705    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b149e027-57c4-47fd-a20b-af8c0969fd08-istiod-ca-cert\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.508958 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.508240    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-podinfo\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.513406 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.513336    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-envoy\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.513406 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.513343    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-token\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.514239 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.514220    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s57nh\" (UniqueName: \"kubernetes.io/projected/b149e027-57c4-47fd-a20b-af8c0969fd08-kube-api-access-s57nh\") pod \"e2e-inf-a-d8b443-openshift-default-559b659988-js6mt\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") " pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.573861 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.573839    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:17.849218 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.849195    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"]
Sep 04 23:48:17.851060 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:48:17.851006    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb149e027_57c4_47fd_a20b_af8c0969fd08.slice/crio-33d47c2fa13309e1f33ad67cf8ad35d09a066662b9b8b2438707c3e1f54bf67e WatchSource:0}: Error finding container 33d47c2fa13309e1f33ad67cf8ad35d09a066662b9b8b2438707c3e1f54bf67e: Status 404 returned error can't find the container with id 33d47c2fa13309e1f33ad67cf8ad35d09a066662b9b8b2438707c3e1f54bf67e
Sep 04 23:48:17.856681 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.856634    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:48:17.856814 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.856712    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:48:17.856814 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:17.856758    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:48:18.262567 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:18.262531    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" event={"ID":"b149e027-57c4-47fd-a20b-af8c0969fd08","Type":"ContainerStarted","Data":"98b10948bec8dfaa64bd8b2e7f10c2312a5628a547f636837660d3b648e8c331"}
Sep 04 23:48:18.262926 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:18.262593    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" event={"ID":"b149e027-57c4-47fd-a20b-af8c0969fd08","Type":"ContainerStarted","Data":"33d47c2fa13309e1f33ad67cf8ad35d09a066662b9b8b2438707c3e1f54bf67e"}
Sep 04 23:48:18.284114 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:18.284064    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" podStartSLOduration=1.284050442 podStartE2EDuration="1.284050442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:48:18.281418839 +0000 UTC m=+680.699237156" watchObservedRunningTime="2026-09-04 23:48:18.284050442 +0000 UTC m=+680.701868759"
Sep 04 23:48:18.574395 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:18.574328    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:18.579451 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:18.579430    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:19.267367 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:19.267334    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:19.268213 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:19.268195    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:20.327766 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:20.327732    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:48:24.811486 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.811454    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"]
Sep 04 23:48:24.815061 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.815034    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.817840 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.817814    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-eef2d470-openshift-default-dockercfg-c4k9p\""
Sep 04 23:48:24.825360 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.825337    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"]
Sep 04 23:48:24.862993 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.862967    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-workload-certs\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.863106 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.863001    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-data\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.863106 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.863020    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-workload-socket\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.863106 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.863087    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-credential-socket\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.863213 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.863116    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j884f\" (UniqueName: \"kubernetes.io/projected/dc518473-fb2e-481f-b59d-c4790f61a96c-kube-api-access-j884f\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.863213 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.863132    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-envoy\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.863213 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.863161    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-podinfo\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.863213 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.863185    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-token\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.863213 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.863199    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc518473-fb2e-481f-b59d-c4790f61a96c-istiod-ca-cert\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.963907 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.963870    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-credential-socket\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.963907 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.963909    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j884f\" (UniqueName: \"kubernetes.io/projected/dc518473-fb2e-481f-b59d-c4790f61a96c-kube-api-access-j884f\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.964168 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.963925    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-envoy\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.964168 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.963973    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-podinfo\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.964168 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.963995    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-token\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.964168 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.964009    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc518473-fb2e-481f-b59d-c4790f61a96c-istiod-ca-cert\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.964168 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.964052    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-workload-certs\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.964168 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.964085    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-data\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.964482 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.964338    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-credential-socket\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.964482 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.964387    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-workload-socket\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.964482 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.964396    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-data\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.964593 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.964491    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-workload-certs\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.964664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.964643    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-workload-socket\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.964781 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.964762    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc518473-fb2e-481f-b59d-c4790f61a96c-istiod-ca-cert\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.966186 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.966160    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-envoy\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.966397 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.966381    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-podinfo\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.971693 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.971666    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-token\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:24.972000 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:24.971982    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j884f\" (UniqueName: \"kubernetes.io/projected/dc518473-fb2e-481f-b59d-c4790f61a96c-kube-api-access-j884f\") pod \"e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") " pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:25.126845 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:25.126774    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:25.402964 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:25.402924    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"]
Sep 04 23:48:25.403579 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:48:25.403554    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc518473_fb2e_481f_b59d_c4790f61a96c.slice/crio-6d0e5d21ea05c7e813cd557354bed6d99f126b394d725b111e924e1bb7041613 WatchSource:0}: Error finding container 6d0e5d21ea05c7e813cd557354bed6d99f126b394d725b111e924e1bb7041613: Status 404 returned error can't find the container with id 6d0e5d21ea05c7e813cd557354bed6d99f126b394d725b111e924e1bb7041613
Sep 04 23:48:25.405736 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:25.405702    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:48:25.405812 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:25.405765    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:48:25.405812 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:25.405792    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:48:26.305697 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:26.305662    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6" event={"ID":"dc518473-fb2e-481f-b59d-c4790f61a96c","Type":"ContainerStarted","Data":"4a9caf17dd9f417936aaac2c11f3d50761e44f30dc3304d3a1d94e6d0099a85d"}
Sep 04 23:48:26.306167 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:26.305715    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6" event={"ID":"dc518473-fb2e-481f-b59d-c4790f61a96c","Type":"ContainerStarted","Data":"6d0e5d21ea05c7e813cd557354bed6d99f126b394d725b111e924e1bb7041613"}
Sep 04 23:48:26.325600 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:26.325548    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6" podStartSLOduration=2.325533266 podStartE2EDuration="2.325533266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:48:26.324068867 +0000 UTC m=+688.741887216" watchObservedRunningTime="2026-09-04 23:48:26.325533266 +0000 UTC m=+688.743351583"
Sep 04 23:48:27.127198 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:27.127159    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:27.131871 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:27.131848    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:27.310813 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:27.310783    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:27.311702 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:27.311683    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:31.164900 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:31.164867    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"]
Sep 04 23:48:31.168362 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:31.168346    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"
Sep 04 23:48:31.171061 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:31.171039    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-3c6688-dockercfg-jqr7s\""
Sep 04 23:48:31.171161 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:31.171086    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-3c6688\""
Sep 04 23:48:31.179202 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:31.179175    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"]
Sep 04 23:48:31.326712 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:31.326677    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mftb9\" (UniqueName: \"kubernetes.io/projected/96f278e3-dc59-4c45-84d2-418f21dd7082-kube-api-access-mftb9\") pod \"payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv\" (UID: \"96f278e3-dc59-4c45-84d2-418f21dd7082\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"
Sep 04 23:48:31.326878 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:31.326800    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f278e3-dc59-4c45-84d2-418f21dd7082-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv\" (UID: \"96f278e3-dc59-4c45-84d2-418f21dd7082\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"
Sep 04 23:48:31.427962 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:31.427857    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mftb9\" (UniqueName: \"kubernetes.io/projected/96f278e3-dc59-4c45-84d2-418f21dd7082-kube-api-access-mftb9\") pod \"payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv\" (UID: \"96f278e3-dc59-4c45-84d2-418f21dd7082\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"
Sep 04 23:48:31.427962 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:31.427955    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f278e3-dc59-4c45-84d2-418f21dd7082-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv\" (UID: \"96f278e3-dc59-4c45-84d2-418f21dd7082\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"
Sep 04 23:48:31.428744 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:31.428720    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f278e3-dc59-4c45-84d2-418f21dd7082-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv\" (UID: \"96f278e3-dc59-4c45-84d2-418f21dd7082\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"
Sep 04 23:48:31.436404 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:31.436382    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mftb9\" (UniqueName: \"kubernetes.io/projected/96f278e3-dc59-4c45-84d2-418f21dd7082-kube-api-access-mftb9\") pod \"payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv\" (UID: \"96f278e3-dc59-4c45-84d2-418f21dd7082\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"
Sep 04 23:48:31.480035 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:31.480008    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"
Sep 04 23:48:31.778757 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:31.778728    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"]
Sep 04 23:48:31.779405 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:48:31.779365    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f278e3_dc59_4c45_84d2_418f21dd7082.slice/crio-cabe2f4b743f570bfc39870b94daf049fa3c27fc2a9093b46fb66240d1155a6f WatchSource:0}: Error finding container cabe2f4b743f570bfc39870b94daf049fa3c27fc2a9093b46fb66240d1155a6f: Status 404 returned error can't find the container with id cabe2f4b743f570bfc39870b94daf049fa3c27fc2a9093b46fb66240d1155a6f
Sep 04 23:48:31.789219 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:31.788050    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 23:48:32.338284 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:32.338250    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv" event={"ID":"96f278e3-dc59-4c45-84d2-418f21dd7082","Type":"ContainerStarted","Data":"cabe2f4b743f570bfc39870b94daf049fa3c27fc2a9093b46fb66240d1155a6f"}
Sep 04 23:48:34.922432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:34.922348    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:48:35.309157 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:35.309118    2557 patch_prober.go:28] interesting pod/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 04 23:48:35.309347 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:35.309180    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 04 23:48:35.360290 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:35.360241    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv" event={"ID":"96f278e3-dc59-4c45-84d2-418f21dd7082","Type":"ContainerStarted","Data":"1fa0f41e8d04bb94606215ff402d397cec04fdaef8f56ac5a2c7f642e4bccbca"}
Sep 04 23:48:35.360486 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:35.360311    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"
Sep 04 23:48:35.378176 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:35.378126    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv" podStartSLOduration=1.718311692 podStartE2EDuration="4.378110744s" totalImagesPullingTime="2.659799052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:48:35.376408252 +0000 UTC m=+697.794226574" watchObservedRunningTime="2026-09-04 23:48:35.378110744 +0000 UTC m=+697.795929061"
Sep 04 23:48:36.800331 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:36.800295    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"]
Sep 04 23:48:36.803879 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:36.803857    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:36.806302 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:36.806286    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-98e5323e-kserve-self-signed-certs\""
Sep 04 23:48:36.814063 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:36.814040    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"]
Sep 04 23:48:36.989377 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:36.989343    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-tmp-dir\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:36.989537 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:36.989399    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-dshm\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:36.989537 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:36.989424    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wd9z\" (UniqueName: \"kubernetes.io/projected/075e7b47-fddd-41f1-a843-b72f05179f51-kube-api-access-9wd9z\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:36.989537 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:36.989495    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-model-cache\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:36.989537 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:36.989531    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-home\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:36.989666 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:36.989548    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/075e7b47-fddd-41f1-a843-b72f05179f51-tls-certs\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:37.090514 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.090432    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-model-cache\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:37.090514 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.090492    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-home\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:37.091882 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.090522    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/075e7b47-fddd-41f1-a843-b72f05179f51-tls-certs\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:37.091882 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.090566    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-tmp-dir\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:37.091882 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.090608    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-dshm\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:37.091882 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.090631    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wd9z\" (UniqueName: \"kubernetes.io/projected/075e7b47-fddd-41f1-a843-b72f05179f51-kube-api-access-9wd9z\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:37.091882 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.090905    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-model-cache\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:37.091882 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.090915    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-home\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:37.091882 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.091142    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-tmp-dir\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:37.093284 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.093254    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-dshm\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:37.093502 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.093473    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/075e7b47-fddd-41f1-a843-b72f05179f51-tls-certs\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:37.098193 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.098171    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wd9z\" (UniqueName: \"kubernetes.io/projected/075e7b47-fddd-41f1-a843-b72f05179f51-kube-api-access-9wd9z\") pod \"e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") " pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:37.116506 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.116200    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:37.374231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.374138    2557 generic.go:356] "Generic (PLEG): container finished" podID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerID="98b10948bec8dfaa64bd8b2e7f10c2312a5628a547f636837660d3b648e8c331" exitCode=0
Sep 04 23:48:37.374231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.374213    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" event={"ID":"b149e027-57c4-47fd-a20b-af8c0969fd08","Type":"ContainerDied","Data":"98b10948bec8dfaa64bd8b2e7f10c2312a5628a547f636837660d3b648e8c331"}
Sep 04 23:48:37.374662 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.374648    2557 scope.go:117] "RemoveContainer" containerID="98b10948bec8dfaa64bd8b2e7f10c2312a5628a547f636837660d3b648e8c331"
Sep 04 23:48:37.375694 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.375661    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:48:37.375809 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.375728    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:48:37.375809 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.375759    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:48:37.418108 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:37.417906    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"]
Sep 04 23:48:38.383584 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:38.383541    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" event={"ID":"b149e027-57c4-47fd-a20b-af8c0969fd08","Type":"ContainerStarted","Data":"a49cf2b360baaf0d1053c84fcdbc5435ab84c20da79131ece8643f77fd2c5770"}
Sep 04 23:48:38.390649 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:38.390608    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k" event={"ID":"075e7b47-fddd-41f1-a843-b72f05179f51","Type":"ContainerStarted","Data":"a64adda5ab83cf193f9ac3ed44452a85308ce27f3e7353d613b834cfc43f5dff"}
Sep 04 23:48:38.390813 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:38.390657    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k" event={"ID":"075e7b47-fddd-41f1-a843-b72f05179f51","Type":"ContainerStarted","Data":"4dad1e34452b096f723e82bd839cd2bf236ebb019c100637bec1fd0d3fdf8933"}
Sep 04 23:48:38.451362 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:38.451300    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k" podStartSLOduration=2.451282537 podStartE2EDuration="2.451282537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:48:38.449182225 +0000 UTC m=+700.867000557" watchObservedRunningTime="2026-09-04 23:48:38.451282537 +0000 UTC m=+700.869100862"
Sep 04 23:48:38.574683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:38.574647    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:38.574859 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:38.574831    2557 patch_prober.go:28] interesting pod/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 04 23:48:38.574923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:38.574899    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 04 23:48:39.575079 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:39.575042    2557 patch_prober.go:28] interesting pod/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 04 23:48:39.575452 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:39.575100    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 04 23:48:40.574768 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:40.574738    2557 patch_prober.go:28] interesting pod/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 04 23:48:40.575025 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:40.574992    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 04 23:48:41.575186 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:41.575152    2557 patch_prober.go:28] interesting pod/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 04 23:48:41.575563 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:41.575205    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 04 23:48:42.574358 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:42.574322    2557 patch_prober.go:28] interesting pod/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 04 23:48:42.574516 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:42.574380    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 04 23:48:43.574321 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:43.574277    2557 patch_prober.go:28] interesting pod/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 04 23:48:43.574703 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:43.574336    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 04 23:48:44.578059 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:44.578030    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:44.578500 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:44.578377    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:44.579015 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:44.578998    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:48:45.618832 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.618799    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"]
Sep 04 23:48:45.627699 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.627672    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.630908 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.630857    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-724e1f-openshift-default-dockercfg-t75gr\""
Sep 04 23:48:45.633614 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.633591    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"]
Sep 04 23:48:45.686663 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.686638    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-envoy\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.686799 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.686666    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-data\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.686799 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.686686    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-workload-certs\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.686799 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.686715    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-token\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.686799 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.686787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-credential-socket\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.686979 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.686835    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-podinfo\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.686979 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.686856    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44c0b19-e574-41e7-96c4-9fab38801f45-istiod-ca-cert\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.686979 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.686883    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqpz4\" (UniqueName: \"kubernetes.io/projected/b44c0b19-e574-41e7-96c4-9fab38801f45-kube-api-access-hqpz4\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.686979 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.686901    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-workload-socket\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.788108 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.788078    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-token\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.790372 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.788122    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-credential-socket\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.790372 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.788156    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-podinfo\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.790372 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.788274    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44c0b19-e574-41e7-96c4-9fab38801f45-istiod-ca-cert\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.790372 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.788330    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqpz4\" (UniqueName: \"kubernetes.io/projected/b44c0b19-e574-41e7-96c4-9fab38801f45-kube-api-access-hqpz4\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.790372 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.788364    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-workload-socket\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.790372 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.788444    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-envoy\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.790372 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.788478    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-data\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.790372 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.788508    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-workload-certs\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.790372 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.788565    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-credential-socket\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.790372 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.788762    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-workload-socket\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.790372 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.788884    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-data\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.790372 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.788904    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-workload-certs\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.790372 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.789040    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44c0b19-e574-41e7-96c4-9fab38801f45-istiod-ca-cert\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.791099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.790717    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-envoy\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.791099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.790781    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-podinfo\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.796673 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.796654    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-token\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.796790 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.796656    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqpz4\" (UniqueName: \"kubernetes.io/projected/b44c0b19-e574-41e7-96c4-9fab38801f45-kube-api-access-hqpz4\") pod \"e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") " pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:45.940573 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:45.940517    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:46.227230 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:46.227185    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"]
Sep 04 23:48:46.229657 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:48:46.229629    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44c0b19_e574_41e7_96c4_9fab38801f45.slice/crio-9e5016f71a81600554b6fa8297c176d04a7337487a83f21ad93e664596480498 WatchSource:0}: Error finding container 9e5016f71a81600554b6fa8297c176d04a7337487a83f21ad93e664596480498: Status 404 returned error can't find the container with id 9e5016f71a81600554b6fa8297c176d04a7337487a83f21ad93e664596480498
Sep 04 23:48:46.231857 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:46.231828    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:48:46.231916 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:46.231902    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:48:46.231976 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:46.231932    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:48:46.368795 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:46.368766    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"
Sep 04 23:48:46.457829 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:46.457790    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7" event={"ID":"b44c0b19-e574-41e7-96c4-9fab38801f45","Type":"ContainerStarted","Data":"7dcb281a389a1f1d7f81e42bb8d720a94cd6b2ea9f501f5fff335482f35b65ab"}
Sep 04 23:48:46.458051 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:46.457851    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7" event={"ID":"b44c0b19-e574-41e7-96c4-9fab38801f45","Type":"ContainerStarted","Data":"9e5016f71a81600554b6fa8297c176d04a7337487a83f21ad93e664596480498"}
Sep 04 23:48:46.459763 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:46.459731    2557 generic.go:356] "Generic (PLEG): container finished" podID="dc518473-fb2e-481f-b59d-c4790f61a96c" containerID="4a9caf17dd9f417936aaac2c11f3d50761e44f30dc3304d3a1d94e6d0099a85d" exitCode=0
Sep 04 23:48:46.459874 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:46.459800    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6" event={"ID":"dc518473-fb2e-481f-b59d-c4790f61a96c","Type":"ContainerDied","Data":"4a9caf17dd9f417936aaac2c11f3d50761e44f30dc3304d3a1d94e6d0099a85d"}
Sep 04 23:48:46.460397 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:46.460381    2557 scope.go:117] "RemoveContainer" containerID="4a9caf17dd9f417936aaac2c11f3d50761e44f30dc3304d3a1d94e6d0099a85d"
Sep 04 23:48:46.461559 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:46.461524    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:48:46.461674 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:46.461632    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:48:46.461674 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:46.461667    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:48:46.477641 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:46.477562    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7" podStartSLOduration=1.477544939 podStartE2EDuration="1.477544939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:48:46.475312185 +0000 UTC m=+708.893130503" watchObservedRunningTime="2026-09-04 23:48:46.477544939 +0000 UTC m=+708.895363258"
Sep 04 23:48:46.940817 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:46.940738    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:47.116952 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:47.116914    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:47.117085 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:47.116974    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:47.128950 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:47.128916    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:47.466990 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:47.466954    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6" event={"ID":"dc518473-fb2e-481f-b59d-c4790f61a96c","Type":"ContainerStarted","Data":"0b990b17714c942318727b6d98d34b671ca7c0ce0a6a1927380f53f25082e9d4"}
Sep 04 23:48:47.478367 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:47.478342    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:48:47.945640 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:47.945560    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:48.127566 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:48.127535    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:48.127730 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:48.127681    2557 patch_prober.go:28] interesting pod/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 04 23:48:48.127775 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:48.127724    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6" podUID="dc518473-fb2e-481f-b59d-c4790f61a96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 04 23:48:48.473132 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:48.473097    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:48.473751 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:48.473730    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:48:49.128115 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:49.128081    2557 patch_prober.go:28] interesting pod/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 04 23:48:49.128505 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:49.128133    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6" podUID="dc518473-fb2e-481f-b59d-c4790f61a96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 04 23:48:50.127628 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:50.127595    2557 patch_prober.go:28] interesting pod/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 04 23:48:50.127789 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:50.127654    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6" podUID="dc518473-fb2e-481f-b59d-c4790f61a96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 04 23:48:51.127508 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:51.127472    2557 patch_prober.go:28] interesting pod/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 04 23:48:51.127899 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:51.127539    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6" podUID="dc518473-fb2e-481f-b59d-c4790f61a96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 04 23:48:51.316606 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:51.316566    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:48:52.127276 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:52.127239    2557 patch_prober.go:28] interesting pod/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 04 23:48:52.127446 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:52.127295    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6" podUID="dc518473-fb2e-481f-b59d-c4790f61a96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 04 23:48:53.040852 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:53.040822    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm"]
Sep 04 23:48:53.044515 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:53.044496    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm"
Sep 04 23:48:53.047603 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:53.047583    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k4f64\""
Sep 04 23:48:53.047828 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:53.047806    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 23:48:53.054374 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:53.054349    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm"]
Sep 04 23:48:53.131930 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:53.131901    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:53.132314 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:53.132294    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:53.132895 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:53.132878    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:48:53.158669 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:53.158636    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c0a77d-77c8-4598-8bfb-347f21356f71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm\" (UID: \"e4c0a77d-77c8-4598-8bfb-347f21356f71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm"
Sep 04 23:48:53.259964 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:53.259912    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c0a77d-77c8-4598-8bfb-347f21356f71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm\" (UID: \"e4c0a77d-77c8-4598-8bfb-347f21356f71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm"
Sep 04 23:48:53.260549 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:53.260524    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c0a77d-77c8-4598-8bfb-347f21356f71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm\" (UID: \"e4c0a77d-77c8-4598-8bfb-347f21356f71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm"
Sep 04 23:48:53.357495 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:53.357415    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm"
Sep 04 23:48:53.633192 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:53.633166    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm"]
Sep 04 23:48:53.633897 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:48:53.633876    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c0a77d_77c8_4598_8bfb_347f21356f71.slice/crio-bed2a989136f25ec070bb3d62cab54b9d3e42faf8394461eacbd2b3c723a8063 WatchSource:0}: Error finding container bed2a989136f25ec070bb3d62cab54b9d3e42faf8394461eacbd2b3c723a8063: Status 404 returned error can't find the container with id bed2a989136f25ec070bb3d62cab54b9d3e42faf8394461eacbd2b3c723a8063
Sep 04 23:48:54.514131 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:54.514100    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm" event={"ID":"e4c0a77d-77c8-4598-8bfb-347f21356f71","Type":"ContainerStarted","Data":"bed2a989136f25ec070bb3d62cab54b9d3e42faf8394461eacbd2b3c723a8063"}
Sep 04 23:48:56.526159 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:56.526131    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm" event={"ID":"e4c0a77d-77c8-4598-8bfb-347f21356f71","Type":"ContainerStarted","Data":"437ec0d4c3a2f96b586a1458f7084d32f0eef8664735d645f49e466d6a0eacda"}
Sep 04 23:48:56.540029 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:56.539985    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm" podStartSLOduration=1.054169584 podStartE2EDuration="3.539970006s" totalImagesPullingTime="2.485800422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:48:56.539551337 +0000 UTC m=+718.957369657" watchObservedRunningTime="2026-09-04 23:48:56.539970006 +0000 UTC m=+718.957788325"
Sep 04 23:48:57.532796 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:57.532760    2557 generic.go:356] "Generic (PLEG): container finished" podID="e4c0a77d-77c8-4598-8bfb-347f21356f71" containerID="437ec0d4c3a2f96b586a1458f7084d32f0eef8664735d645f49e466d6a0eacda" exitCode=0
Sep 04 23:48:57.533187 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:57.532838    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm" event={"ID":"e4c0a77d-77c8-4598-8bfb-347f21356f71","Type":"ContainerDied","Data":"437ec0d4c3a2f96b586a1458f7084d32f0eef8664735d645f49e466d6a0eacda"}
Sep 04 23:48:58.663744 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:58.663719    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm"
Sep 04 23:48:58.705831 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:58.705794    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c0a77d-77c8-4598-8bfb-347f21356f71-maas-api-service-ca\") pod \"e4c0a77d-77c8-4598-8bfb-347f21356f71\" (UID: \"e4c0a77d-77c8-4598-8bfb-347f21356f71\") "
Sep 04 23:48:58.706389 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:58.706335    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c0a77d-77c8-4598-8bfb-347f21356f71-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e4c0a77d-77c8-4598-8bfb-347f21356f71" (UID: "e4c0a77d-77c8-4598-8bfb-347f21356f71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:48:58.807786 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:58.807672    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c0a77d-77c8-4598-8bfb-347f21356f71-maas-api-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:48:59.546255 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:59.546227    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm"
Sep 04 23:48:59.546255 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:59.546235    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm" event={"ID":"e4c0a77d-77c8-4598-8bfb-347f21356f71","Type":"ContainerDied","Data":"bed2a989136f25ec070bb3d62cab54b9d3e42faf8394461eacbd2b3c723a8063"}
Sep 04 23:48:59.546468 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:48:59.546283    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bed2a989136f25ec070bb3d62cab54b9d3e42faf8394461eacbd2b3c723a8063"
Sep 04 23:49:06.811991 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:06.811956    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:49:07.594598 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:07.592125    2557 generic.go:356] "Generic (PLEG): container finished" podID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerID="7dcb281a389a1f1d7f81e42bb8d720a94cd6b2ea9f501f5fff335482f35b65ab" exitCode=0
Sep 04 23:49:07.594598 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:07.592233    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7" event={"ID":"b44c0b19-e574-41e7-96c4-9fab38801f45","Type":"ContainerDied","Data":"7dcb281a389a1f1d7f81e42bb8d720a94cd6b2ea9f501f5fff335482f35b65ab"}
Sep 04 23:49:07.594598 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:07.592674    2557 scope.go:117] "RemoveContainer" containerID="7dcb281a389a1f1d7f81e42bb8d720a94cd6b2ea9f501f5fff335482f35b65ab"
Sep 04 23:49:07.595375 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:07.595130    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:49:07.595375 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:07.595226    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:49:07.595375 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:07.595276    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:49:08.599142 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:08.599113    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7" event={"ID":"b44c0b19-e574-41e7-96c4-9fab38801f45","Type":"ContainerStarted","Data":"99b2eeece76335e8faeb9c764fbc801d3ce8e0e19c5cd39885054970ba8d5228"}
Sep 04 23:49:08.941516 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:08.941415    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:49:08.941700 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:08.941670    2557 patch_prober.go:28] interesting pod/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 04 23:49:08.941794 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:08.941737    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7" podUID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 04 23:49:09.766305 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:09.766272    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm"]
Sep 04 23:49:09.768422 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:09.768400    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eef2d470-vfxnm"]
Sep 04 23:49:09.918540 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:09.918504    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"]
Sep 04 23:49:09.918785 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:09.918756    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6" podUID="dc518473-fb2e-481f-b59d-c4790f61a96c" containerName="istio-proxy" containerID="cri-o://0b990b17714c942318727b6d98d34b671ca7c0ce0a6a1927380f53f25082e9d4" gracePeriod=30
Sep 04 23:49:09.941197 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:09.941156    2557 patch_prober.go:28] interesting pod/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 04 23:49:09.941390 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:09.941203    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7" podUID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 04 23:49:10.078654 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.078575    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4c0a77d-77c8-4598-8bfb-347f21356f71" path="/var/lib/kubelet/pods/e4c0a77d-77c8-4598-8bfb-347f21356f71/volumes"
Sep 04 23:49:10.831609 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.831576    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"]
Sep 04 23:49:10.832036 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.832019    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c0a77d-77c8-4598-8bfb-347f21356f71" containerName="revoke-keys"
Sep 04 23:49:10.832094 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.832040    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c0a77d-77c8-4598-8bfb-347f21356f71" containerName="revoke-keys"
Sep 04 23:49:10.832131 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.832107    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c0a77d-77c8-4598-8bfb-347f21356f71" containerName="revoke-keys"
Sep 04 23:49:10.834120 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.834102    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:10.838500 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.838472    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8a8c53fa-gw-openshift-default-dockercfg-47g6v\""
Sep 04 23:49:10.858062 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.858037    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"]
Sep 04 23:49:10.927281 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.926767    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-podinfo\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:10.927281 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.926875    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-workload-socket\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:10.927281 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.926910    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-credential-socket\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:10.927281 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.926952    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-workload-certs\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:10.927281 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.927006    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-data\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:10.927281 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.927032    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istiod-ca-cert\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:10.927281 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.927058    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-envoy\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:10.927281 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.927090    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-token\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:10.927281 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.927135    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww89j\" (UniqueName: \"kubernetes.io/projected/57d41641-3d08-45d2-b4a2-124cbe13d0b8-kube-api-access-ww89j\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:10.941010 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.940973    2557 patch_prober.go:28] interesting pod/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 04 23:49:10.941183 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:10.941034    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7" podUID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 04 23:49:11.028508 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.028472    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-podinfo\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.028672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.028567    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-workload-socket\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.028672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.028600    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-credential-socket\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.028672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.028622    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-workload-certs\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.029924 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.029897    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-workload-socket\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.030051 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.029962    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-data\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.030051 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.030021    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istiod-ca-cert\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.030156 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.030053    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-envoy\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.030156 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.030097    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-token\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.030225 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.030151    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww89j\" (UniqueName: \"kubernetes.io/projected/57d41641-3d08-45d2-b4a2-124cbe13d0b8-kube-api-access-ww89j\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.030274 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.030247    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-credential-socket\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.030274 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.029904    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-data\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.030339 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.030299    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-workload-certs\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.031564 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.031538    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istiod-ca-cert\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.033016 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.032989    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-podinfo\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.036286 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.036257    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-envoy\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.042683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.042662    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww89j\" (UniqueName: \"kubernetes.io/projected/57d41641-3d08-45d2-b4a2-124cbe13d0b8-kube-api-access-ww89j\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.043089 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.043072    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-token\") pod \"e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") " pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.146783 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.146697    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:11.447275 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.447251    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"]
Sep 04 23:49:11.448887 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:49:11.448835    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d41641_3d08_45d2_b4a2_124cbe13d0b8.slice/crio-f1140a4ee10f9cfae349b70ff4aba579425add2211a4497d2f486d87b0ad34ce WatchSource:0}: Error finding container f1140a4ee10f9cfae349b70ff4aba579425add2211a4497d2f486d87b0ad34ce: Status 404 returned error can't find the container with id f1140a4ee10f9cfae349b70ff4aba579425add2211a4497d2f486d87b0ad34ce
Sep 04 23:49:11.451437 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.451398    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:49:11.451540 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.451480    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:49:11.451540 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.451524    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:49:11.629999 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.629965    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr" event={"ID":"57d41641-3d08-45d2-b4a2-124cbe13d0b8","Type":"ContainerStarted","Data":"a40a619892aa0baa108c302a8814eb71bed1f482d0f07d1ee36b8a6d9e1bbef8"}
Sep 04 23:49:11.630182 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.630018    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr" event={"ID":"57d41641-3d08-45d2-b4a2-124cbe13d0b8","Type":"ContainerStarted","Data":"f1140a4ee10f9cfae349b70ff4aba579425add2211a4497d2f486d87b0ad34ce"}
Sep 04 23:49:11.655388 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.655338    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr" podStartSLOduration=1.655324112 podStartE2EDuration="1.655324112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:49:11.652977186 +0000 UTC m=+734.070795508" watchObservedRunningTime="2026-09-04 23:49:11.655324112 +0000 UTC m=+734.073142429"
Sep 04 23:49:11.941091 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.941006    2557 patch_prober.go:28] interesting pod/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 04 23:49:11.941541 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:11.941089    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7" podUID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 04 23:49:12.147380 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:12.147346    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:12.152335 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:12.152314    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:12.635717 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:12.635690    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:12.636604 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:12.636585    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:12.941569 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:12.941489    2557 patch_prober.go:28] interesting pod/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 04 23:49:12.941569 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:12.941538    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7" podUID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 04 23:49:13.942577 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:13.941771    2557 patch_prober.go:28] interesting pod/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 04 23:49:13.942577 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:13.941835    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7" podUID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 04 23:49:14.945635 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:14.945612    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:49:14.946048 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:14.946031    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:49:14.946683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:14.946664    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:49:15.176399 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.176374    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:49:15.269602 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.269577    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-data\") pod \"dc518473-fb2e-481f-b59d-c4790f61a96c\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") "
Sep 04 23:49:15.269764 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.269624    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j884f\" (UniqueName: \"kubernetes.io/projected/dc518473-fb2e-481f-b59d-c4790f61a96c-kube-api-access-j884f\") pod \"dc518473-fb2e-481f-b59d-c4790f61a96c\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") "
Sep 04 23:49:15.269764 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.269648    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-credential-socket\") pod \"dc518473-fb2e-481f-b59d-c4790f61a96c\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") "
Sep 04 23:49:15.269764 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.269666    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-envoy\") pod \"dc518473-fb2e-481f-b59d-c4790f61a96c\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") "
Sep 04 23:49:15.269764 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.269707    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-workload-socket\") pod \"dc518473-fb2e-481f-b59d-c4790f61a96c\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") "
Sep 04 23:49:15.269764 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.269728    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-token\") pod \"dc518473-fb2e-481f-b59d-c4790f61a96c\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") "
Sep 04 23:49:15.269764 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.269744    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc518473-fb2e-481f-b59d-c4790f61a96c-istiod-ca-cert\") pod \"dc518473-fb2e-481f-b59d-c4790f61a96c\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") "
Sep 04 23:49:15.270136 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.269793    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-workload-certs\") pod \"dc518473-fb2e-481f-b59d-c4790f61a96c\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") "
Sep 04 23:49:15.270136 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.269821    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-podinfo\") pod \"dc518473-fb2e-481f-b59d-c4790f61a96c\" (UID: \"dc518473-fb2e-481f-b59d-c4790f61a96c\") "
Sep 04 23:49:15.270136 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.269883    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-data" (OuterVolumeSpecName: "istio-data") pod "dc518473-fb2e-481f-b59d-c4790f61a96c" (UID: "dc518473-fb2e-481f-b59d-c4790f61a96c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:49:15.270136 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.269891    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dc518473-fb2e-481f-b59d-c4790f61a96c" (UID: "dc518473-fb2e-481f-b59d-c4790f61a96c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:49:15.270136 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.270060    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dc518473-fb2e-481f-b59d-c4790f61a96c" (UID: "dc518473-fb2e-481f-b59d-c4790f61a96c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:49:15.270136 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.270132    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dc518473-fb2e-481f-b59d-c4790f61a96c" (UID: "dc518473-fb2e-481f-b59d-c4790f61a96c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:49:15.270440 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.270204    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:15.270440 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.270209    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc518473-fb2e-481f-b59d-c4790f61a96c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dc518473-fb2e-481f-b59d-c4790f61a96c" (UID: "dc518473-fb2e-481f-b59d-c4790f61a96c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:49:15.270440 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.270222    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:15.270440 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.270236    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:15.270440 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.270247    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:15.271933 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.271901    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dc518473-fb2e-481f-b59d-c4790f61a96c" (UID: "dc518473-fb2e-481f-b59d-c4790f61a96c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:49:15.272078 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.271985    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dc518473-fb2e-481f-b59d-c4790f61a96c" (UID: "dc518473-fb2e-481f-b59d-c4790f61a96c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:49:15.272078 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.271991    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc518473-fb2e-481f-b59d-c4790f61a96c-kube-api-access-j884f" (OuterVolumeSpecName: "kube-api-access-j884f") pod "dc518473-fb2e-481f-b59d-c4790f61a96c" (UID: "dc518473-fb2e-481f-b59d-c4790f61a96c"). InnerVolumeSpecName "kube-api-access-j884f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:49:15.272078 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.272002    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-token" (OuterVolumeSpecName: "istio-token") pod "dc518473-fb2e-481f-b59d-c4790f61a96c" (UID: "dc518473-fb2e-481f-b59d-c4790f61a96c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:49:15.371258 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.371235    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j884f\" (UniqueName: \"kubernetes.io/projected/dc518473-fb2e-481f-b59d-c4790f61a96c-kube-api-access-j884f\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:15.371258 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.371255    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:15.371392 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.371265    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:15.371392 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.371274    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc518473-fb2e-481f-b59d-c4790f61a96c-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:15.371392 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.371282    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc518473-fb2e-481f-b59d-c4790f61a96c-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:15.654575 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.654502    2557 generic.go:356] "Generic (PLEG): container finished" podID="dc518473-fb2e-481f-b59d-c4790f61a96c" containerID="0b990b17714c942318727b6d98d34b671ca7c0ce0a6a1927380f53f25082e9d4" exitCode=0
Sep 04 23:49:15.654575 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.654561    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"
Sep 04 23:49:15.654720 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.654580    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6" event={"ID":"dc518473-fb2e-481f-b59d-c4790f61a96c","Type":"ContainerDied","Data":"0b990b17714c942318727b6d98d34b671ca7c0ce0a6a1927380f53f25082e9d4"}
Sep 04 23:49:15.654720 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.654661    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6" event={"ID":"dc518473-fb2e-481f-b59d-c4790f61a96c","Type":"ContainerDied","Data":"6d0e5d21ea05c7e813cd557354bed6d99f126b394d725b111e924e1bb7041613"}
Sep 04 23:49:15.654720 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.654689    2557 scope.go:117] "RemoveContainer" containerID="0b990b17714c942318727b6d98d34b671ca7c0ce0a6a1927380f53f25082e9d4"
Sep 04 23:49:15.664737 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.664722    2557 scope.go:117] "RemoveContainer" containerID="4a9caf17dd9f417936aaac2c11f3d50761e44f30dc3304d3a1d94e6d0099a85d"
Sep 04 23:49:15.672742 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.672726    2557 scope.go:117] "RemoveContainer" containerID="0b990b17714c942318727b6d98d34b671ca7c0ce0a6a1927380f53f25082e9d4"
Sep 04 23:49:15.673055 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:49:15.673030    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b990b17714c942318727b6d98d34b671ca7c0ce0a6a1927380f53f25082e9d4\": container with ID starting with 0b990b17714c942318727b6d98d34b671ca7c0ce0a6a1927380f53f25082e9d4 not found: ID does not exist" containerID="0b990b17714c942318727b6d98d34b671ca7c0ce0a6a1927380f53f25082e9d4"
Sep 04 23:49:15.673131 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.673077    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b990b17714c942318727b6d98d34b671ca7c0ce0a6a1927380f53f25082e9d4"} err="failed to get container status \"0b990b17714c942318727b6d98d34b671ca7c0ce0a6a1927380f53f25082e9d4\": rpc error: code = NotFound desc = could not find container \"0b990b17714c942318727b6d98d34b671ca7c0ce0a6a1927380f53f25082e9d4\": container with ID starting with 0b990b17714c942318727b6d98d34b671ca7c0ce0a6a1927380f53f25082e9d4 not found: ID does not exist"
Sep 04 23:49:15.673131 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.673109    2557 scope.go:117] "RemoveContainer" containerID="4a9caf17dd9f417936aaac2c11f3d50761e44f30dc3304d3a1d94e6d0099a85d"
Sep 04 23:49:15.673524 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:49:15.673493    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9caf17dd9f417936aaac2c11f3d50761e44f30dc3304d3a1d94e6d0099a85d\": container with ID starting with 4a9caf17dd9f417936aaac2c11f3d50761e44f30dc3304d3a1d94e6d0099a85d not found: ID does not exist" containerID="4a9caf17dd9f417936aaac2c11f3d50761e44f30dc3304d3a1d94e6d0099a85d"
Sep 04 23:49:15.673653 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.673629    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9caf17dd9f417936aaac2c11f3d50761e44f30dc3304d3a1d94e6d0099a85d"} err="failed to get container status \"4a9caf17dd9f417936aaac2c11f3d50761e44f30dc3304d3a1d94e6d0099a85d\": rpc error: code = NotFound desc = could not find container \"4a9caf17dd9f417936aaac2c11f3d50761e44f30dc3304d3a1d94e6d0099a85d\": container with ID starting with 4a9caf17dd9f417936aaac2c11f3d50761e44f30dc3304d3a1d94e6d0099a85d not found: ID does not exist"
Sep 04 23:49:15.676304 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.676284    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"]
Sep 04 23:49:15.680147 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:15.680126    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-eef2d470-openshift-default-5f799c9bbb-ffrb6"]
Sep 04 23:49:16.080362 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:16.080333    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc518473-fb2e-481f-b59d-c4790f61a96c" path="/var/lib/kubelet/pods/dc518473-fb2e-481f-b59d-c4790f61a96c/volumes"
Sep 04 23:49:24.313702 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:24.313668    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:49:31.414351 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.414313    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"]
Sep 04 23:49:31.414904 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.414885    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc518473-fb2e-481f-b59d-c4790f61a96c" containerName="istio-proxy"
Sep 04 23:49:31.415063 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.415049    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc518473-fb2e-481f-b59d-c4790f61a96c" containerName="istio-proxy"
Sep 04 23:49:31.415348 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.415334    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc518473-fb2e-481f-b59d-c4790f61a96c" containerName="istio-proxy"
Sep 04 23:49:31.415441 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.415432    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc518473-fb2e-481f-b59d-c4790f61a96c" containerName="istio-proxy"
Sep 04 23:49:31.418721 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.418698    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.423263 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.423238    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-98e5323e-kserve-self-signed-certs\""
Sep 04 23:49:31.439826 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.439798    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"]
Sep 04 23:49:31.508256 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.508220    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-model-cache\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.508447 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.508355    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4tjm\" (UniqueName: \"kubernetes.io/projected/5b34a041-67c5-4b1f-865b-266e3e727d7d-kube-api-access-r4tjm\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.508447 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.508403    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-dshm\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.508584 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.508514    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-tmp-dir\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.508584 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.508557    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-home\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.508676 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.508657    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b34a041-67c5-4b1f-865b-266e3e727d7d-tls-certs\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.609875 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.609830    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-model-cache\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.610085 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.609959    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4tjm\" (UniqueName: \"kubernetes.io/projected/5b34a041-67c5-4b1f-865b-266e3e727d7d-kube-api-access-r4tjm\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.610085 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.609991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-dshm\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.610085 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.610054    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-tmp-dir\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.610267 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.610089    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-home\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.610267 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.610142    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b34a041-67c5-4b1f-865b-266e3e727d7d-tls-certs\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.611219 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.611189    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-tmp-dir\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.612241 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.611612    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-model-cache\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.612378 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.612169    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-home\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.616236 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.616212    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-dshm\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.616633 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.616609    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b34a041-67c5-4b1f-865b-266e3e727d7d-tls-certs\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.624080 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.624053    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4tjm\" (UniqueName: \"kubernetes.io/projected/5b34a041-67c5-4b1f-865b-266e3e727d7d-kube-api-access-r4tjm\") pod \"e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") " pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:31.745332 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:31.745292    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:32.040920 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:32.040889    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"]
Sep 04 23:49:32.041822 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:49:32.041798    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b34a041_67c5_4b1f_865b_266e3e727d7d.slice/crio-a8daa7f56fd137a6b3ff19851f2291337222c213718a321bbab2ea2f4e41e0e9 WatchSource:0}: Error finding container a8daa7f56fd137a6b3ff19851f2291337222c213718a321bbab2ea2f4e41e0e9: Status 404 returned error can't find the container with id a8daa7f56fd137a6b3ff19851f2291337222c213718a321bbab2ea2f4e41e0e9
Sep 04 23:49:32.757652 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:32.757613    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d" event={"ID":"5b34a041-67c5-4b1f-865b-266e3e727d7d","Type":"ContainerStarted","Data":"8cf89cb20c1265e7d08f343c08e7c1decee897eac7d023a6dc71905580a0b691"}
Sep 04 23:49:32.758095 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:32.757659    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d" event={"ID":"5b34a041-67c5-4b1f-865b-266e3e727d7d","Type":"ContainerStarted","Data":"a8daa7f56fd137a6b3ff19851f2291337222c213718a321bbab2ea2f4e41e0e9"}
Sep 04 23:49:32.775070 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:32.775021    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d" podStartSLOduration=1.775005175 podStartE2EDuration="1.775005175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:49:32.772492968 +0000 UTC m=+755.190311291" watchObservedRunningTime="2026-09-04 23:49:32.775005175 +0000 UTC m=+755.192823493"
Sep 04 23:49:37.985780 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:37.985748    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq"]
Sep 04 23:49:37.986252 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:37.986180    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc518473-fb2e-481f-b59d-c4790f61a96c" containerName="istio-proxy"
Sep 04 23:49:37.986252 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:37.986193    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc518473-fb2e-481f-b59d-c4790f61a96c" containerName="istio-proxy"
Sep 04 23:49:37.989446 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:37.989429    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq"
Sep 04 23:49:37.992021 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:37.991996    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 23:49:37.992158 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:37.991995    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k4f64\""
Sep 04 23:49:37.998790 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:37.998065    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq"]
Sep 04 23:49:38.173429 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:38.173374    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1042cb8-6c77-4aa4-a44a-31d324ecb0f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq\" (UID: \"b1042cb8-6c77-4aa4-a44a-31d324ecb0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq"
Sep 04 23:49:38.274958 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:38.274828    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1042cb8-6c77-4aa4-a44a-31d324ecb0f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq\" (UID: \"b1042cb8-6c77-4aa4-a44a-31d324ecb0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq"
Sep 04 23:49:38.275494 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:38.275471    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1042cb8-6c77-4aa4-a44a-31d324ecb0f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq\" (UID: \"b1042cb8-6c77-4aa4-a44a-31d324ecb0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq"
Sep 04 23:49:38.302795 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:38.302764    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq"
Sep 04 23:49:38.577685 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:38.577650    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq"]
Sep 04 23:49:38.579193 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:49:38.579160    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1042cb8_6c77_4aa4_a44a_31d324ecb0f7.slice/crio-278b8928aab20ba5132bbb144563d804d711406984de3ac59a7fc40bb4edcbac WatchSource:0}: Error finding container 278b8928aab20ba5132bbb144563d804d711406984de3ac59a7fc40bb4edcbac: Status 404 returned error can't find the container with id 278b8928aab20ba5132bbb144563d804d711406984de3ac59a7fc40bb4edcbac
Sep 04 23:49:38.791838 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:38.791809    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq" event={"ID":"b1042cb8-6c77-4aa4-a44a-31d324ecb0f7","Type":"ContainerStarted","Data":"278b8928aab20ba5132bbb144563d804d711406984de3ac59a7fc40bb4edcbac"}
Sep 04 23:49:38.811687 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:38.811660    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:49:39.798469 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:39.798436    2557 generic.go:356] "Generic (PLEG): container finished" podID="b1042cb8-6c77-4aa4-a44a-31d324ecb0f7" containerID="a15260ea07f10c4b0c7e90d378d4dfc60a2196abe1a5f98b6671b0d91c60d5fd" exitCode=0
Sep 04 23:49:39.798841 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:39.798518    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq" event={"ID":"b1042cb8-6c77-4aa4-a44a-31d324ecb0f7","Type":"ContainerDied","Data":"a15260ea07f10c4b0c7e90d378d4dfc60a2196abe1a5f98b6671b0d91c60d5fd"}
Sep 04 23:49:39.988495 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:49:39.988466    2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d41641_3d08_45d2_b4a2_124cbe13d0b8.slice/crio-conmon-a40a619892aa0baa108c302a8814eb71bed1f482d0f07d1ee36b8a6d9e1bbef8.scope\": RecentStats: unable to find data in memory cache]"
Sep 04 23:49:40.804884 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:40.804850    2557 generic.go:356] "Generic (PLEG): container finished" podID="57d41641-3d08-45d2-b4a2-124cbe13d0b8" containerID="a40a619892aa0baa108c302a8814eb71bed1f482d0f07d1ee36b8a6d9e1bbef8" exitCode=0
Sep 04 23:49:40.805259 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:40.804924    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr" event={"ID":"57d41641-3d08-45d2-b4a2-124cbe13d0b8","Type":"ContainerDied","Data":"a40a619892aa0baa108c302a8814eb71bed1f482d0f07d1ee36b8a6d9e1bbef8"}
Sep 04 23:49:40.805423 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:40.805403    2557 scope.go:117] "RemoveContainer" containerID="a40a619892aa0baa108c302a8814eb71bed1f482d0f07d1ee36b8a6d9e1bbef8"
Sep 04 23:49:40.806582 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:40.806549    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:49:40.806714 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:40.806618    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:49:40.806714 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:40.806655    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:49:40.954761 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:40.954735    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq"
Sep 04 23:49:41.100047 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:41.099969    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1042cb8-6c77-4aa4-a44a-31d324ecb0f7-maas-api-service-ca\") pod \"b1042cb8-6c77-4aa4-a44a-31d324ecb0f7\" (UID: \"b1042cb8-6c77-4aa4-a44a-31d324ecb0f7\") "
Sep 04 23:49:41.100323 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:41.100300    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1042cb8-6c77-4aa4-a44a-31d324ecb0f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1042cb8-6c77-4aa4-a44a-31d324ecb0f7" (UID: "b1042cb8-6c77-4aa4-a44a-31d324ecb0f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:49:41.201158 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:41.201118    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1042cb8-6c77-4aa4-a44a-31d324ecb0f7-maas-api-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:41.746397 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:41.746359    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:41.746397 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:41.746407    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:41.759184 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:41.759156    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:41.818072 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:41.818031    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr" event={"ID":"57d41641-3d08-45d2-b4a2-124cbe13d0b8","Type":"ContainerStarted","Data":"582b2fe0449ccbc735f21dede096017c33abffc56813475ce6f53ed1654788c8"}
Sep 04 23:49:41.819650 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:41.819624    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq" event={"ID":"b1042cb8-6c77-4aa4-a44a-31d324ecb0f7","Type":"ContainerDied","Data":"278b8928aab20ba5132bbb144563d804d711406984de3ac59a7fc40bb4edcbac"}
Sep 04 23:49:41.819650 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:41.819647    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq"
Sep 04 23:49:41.819813 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:41.819671    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="278b8928aab20ba5132bbb144563d804d711406984de3ac59a7fc40bb4edcbac"
Sep 04 23:49:41.831442 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:41.831419    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:49:42.147881 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:42.147793    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:42.149021 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:42.148994    2557 patch_prober.go:28] interesting pod/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 04 23:49:42.149145 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:42.149049    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr" podUID="57d41641-3d08-45d2-b4a2-124cbe13d0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 04 23:49:42.635931 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:42.635899    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:43.147456 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:43.147424    2557 patch_prober.go:28] interesting pod/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 04 23:49:43.147817 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:43.147471    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr" podUID="57d41641-3d08-45d2-b4a2-124cbe13d0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 04 23:49:44.147525 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:44.147490    2557 patch_prober.go:28] interesting pod/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 04 23:49:44.147889 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:44.147549    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr" podUID="57d41641-3d08-45d2-b4a2-124cbe13d0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 04 23:49:44.863757 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:44.863728    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq"]
Sep 04 23:49:44.868830 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:44.868807    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a8c53fa-vngkq"]
Sep 04 23:49:45.003112 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:45.003076    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"]
Sep 04 23:49:45.003556 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:45.003523    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr" podUID="57d41641-3d08-45d2-b4a2-124cbe13d0b8" containerName="istio-proxy" containerID="cri-o://582b2fe0449ccbc735f21dede096017c33abffc56813475ce6f53ed1654788c8" gracePeriod=30
Sep 04 23:49:46.078893 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:46.078859    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1042cb8-6c77-4aa4-a44a-31d324ecb0f7" path="/var/lib/kubelet/pods/b1042cb8-6c77-4aa4-a44a-31d324ecb0f7/volumes"
Sep 04 23:49:49.550313 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:49.550271    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"]
Sep 04 23:49:49.550676 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:49.550663    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1042cb8-6c77-4aa4-a44a-31d324ecb0f7" containerName="revoke-keys"
Sep 04 23:49:49.550721 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:49.550678    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1042cb8-6c77-4aa4-a44a-31d324ecb0f7" containerName="revoke-keys"
Sep 04 23:49:49.550766 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:49.550756    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1042cb8-6c77-4aa4-a44a-31d324ecb0f7" containerName="revoke-keys"
Sep 04 23:49:49.553396 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:49.553379    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"
Sep 04 23:49:49.555926 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:49.555905    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b52c7b74-dockercfg-xxpg8\""
Sep 04 23:49:49.556072 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:49.556019    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b52c7b74\""
Sep 04 23:49:49.564050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:49.564025    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"]
Sep 04 23:49:49.576216 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:49.576185    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0dc2ad-dd0e-42be-a992-e1b39596d744-plugins-config-volume\") pod \"payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt\" (UID: \"fd0dc2ad-dd0e-42be-a992-e1b39596d744\") " pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"
Sep 04 23:49:49.576363 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:49.576225    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5n8l\" (UniqueName: \"kubernetes.io/projected/fd0dc2ad-dd0e-42be-a992-e1b39596d744-kube-api-access-w5n8l\") pod \"payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt\" (UID: \"fd0dc2ad-dd0e-42be-a992-e1b39596d744\") " pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"
Sep 04 23:49:49.677335 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:49.677297    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0dc2ad-dd0e-42be-a992-e1b39596d744-plugins-config-volume\") pod \"payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt\" (UID: \"fd0dc2ad-dd0e-42be-a992-e1b39596d744\") " pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"
Sep 04 23:49:49.677498 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:49.677357    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5n8l\" (UniqueName: \"kubernetes.io/projected/fd0dc2ad-dd0e-42be-a992-e1b39596d744-kube-api-access-w5n8l\") pod \"payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt\" (UID: \"fd0dc2ad-dd0e-42be-a992-e1b39596d744\") " pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"
Sep 04 23:49:49.678188 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:49.678169    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0dc2ad-dd0e-42be-a992-e1b39596d744-plugins-config-volume\") pod \"payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt\" (UID: \"fd0dc2ad-dd0e-42be-a992-e1b39596d744\") " pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"
Sep 04 23:49:49.685043 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:49.685021    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5n8l\" (UniqueName: \"kubernetes.io/projected/fd0dc2ad-dd0e-42be-a992-e1b39596d744-kube-api-access-w5n8l\") pod \"payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt\" (UID: \"fd0dc2ad-dd0e-42be-a992-e1b39596d744\") " pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"
Sep 04 23:49:49.864992 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:49.864913    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"
Sep 04 23:49:50.132843 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:50.132817    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"]
Sep 04 23:49:50.134184 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:49:50.134159    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0dc2ad_dd0e_42be_a992_e1b39596d744.slice/crio-db3b8d52cb31bd5ec89e7ac3d71fb2d85fab0ec5a7cf4744f68b1ed4f8152c57 WatchSource:0}: Error finding container db3b8d52cb31bd5ec89e7ac3d71fb2d85fab0ec5a7cf4744f68b1ed4f8152c57: Status 404 returned error can't find the container with id db3b8d52cb31bd5ec89e7ac3d71fb2d85fab0ec5a7cf4744f68b1ed4f8152c57
Sep 04 23:49:50.875537 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:50.875502    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt" event={"ID":"fd0dc2ad-dd0e-42be-a992-e1b39596d744","Type":"ContainerStarted","Data":"5d4c9104b5858dce03cdb862a35d335764d1806c657d65c4295b6708b755d6d4"}
Sep 04 23:49:50.875884 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:50.875565    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"
Sep 04 23:49:50.875884 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:50.875590    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt" event={"ID":"fd0dc2ad-dd0e-42be-a992-e1b39596d744","Type":"ContainerStarted","Data":"db3b8d52cb31bd5ec89e7ac3d71fb2d85fab0ec5a7cf4744f68b1ed4f8152c57"}
Sep 04 23:49:50.892840 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:50.892793    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt" podStartSLOduration=1.892778515 podStartE2EDuration="1.892778515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:49:50.890726186 +0000 UTC m=+773.308544506" watchObservedRunningTime="2026-09-04 23:49:50.892778515 +0000 UTC m=+773.310596875"
Sep 04 23:49:52.158835 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.158809    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:52.197076 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197048    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-token\") pod \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") "
Sep 04 23:49:52.197186 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197091    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-envoy\") pod \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") "
Sep 04 23:49:52.197186 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197112    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww89j\" (UniqueName: \"kubernetes.io/projected/57d41641-3d08-45d2-b4a2-124cbe13d0b8-kube-api-access-ww89j\") pod \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") "
Sep 04 23:49:52.197186 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197152    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-workload-socket\") pod \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") "
Sep 04 23:49:52.197186 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197175    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istiod-ca-cert\") pod \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") "
Sep 04 23:49:52.197376 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197208    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-workload-certs\") pod \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") "
Sep 04 23:49:52.197376 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197345    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-data\") pod \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") "
Sep 04 23:49:52.197482 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197435    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-podinfo\") pod \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") "
Sep 04 23:49:52.197482 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197475    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-credential-socket\") pod \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\" (UID: \"57d41641-3d08-45d2-b4a2-124cbe13d0b8\") "
Sep 04 23:49:52.197579 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197517    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "57d41641-3d08-45d2-b4a2-124cbe13d0b8" (UID: "57d41641-3d08-45d2-b4a2-124cbe13d0b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:49:52.197635 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197603    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "57d41641-3d08-45d2-b4a2-124cbe13d0b8" (UID: "57d41641-3d08-45d2-b4a2-124cbe13d0b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:49:52.197717 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197697    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "57d41641-3d08-45d2-b4a2-124cbe13d0b8" (UID: "57d41641-3d08-45d2-b4a2-124cbe13d0b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:49:52.197717 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197675    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "57d41641-3d08-45d2-b4a2-124cbe13d0b8" (UID: "57d41641-3d08-45d2-b4a2-124cbe13d0b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:49:52.197912 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197850    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:52.197912 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197874    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:52.197912 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197879    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "57d41641-3d08-45d2-b4a2-124cbe13d0b8" (UID: "57d41641-3d08-45d2-b4a2-124cbe13d0b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:49:52.197912 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197888    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:52.198198 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.197920    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:52.199510 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.199482    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "57d41641-3d08-45d2-b4a2-124cbe13d0b8" (UID: "57d41641-3d08-45d2-b4a2-124cbe13d0b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:49:52.199510 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.199494    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d41641-3d08-45d2-b4a2-124cbe13d0b8-kube-api-access-ww89j" (OuterVolumeSpecName: "kube-api-access-ww89j") pod "57d41641-3d08-45d2-b4a2-124cbe13d0b8" (UID: "57d41641-3d08-45d2-b4a2-124cbe13d0b8"). InnerVolumeSpecName "kube-api-access-ww89j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:49:52.199635 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.199525    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "57d41641-3d08-45d2-b4a2-124cbe13d0b8" (UID: "57d41641-3d08-45d2-b4a2-124cbe13d0b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:49:52.199635 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.199619    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "57d41641-3d08-45d2-b4a2-124cbe13d0b8" (UID: "57d41641-3d08-45d2-b4a2-124cbe13d0b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:49:52.299318 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.299263    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:52.299318 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.299284    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:52.299318 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.299295    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:52.299318 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.299303    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d41641-3d08-45d2-b4a2-124cbe13d0b8-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:52.299318 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.299311    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww89j\" (UniqueName: \"kubernetes.io/projected/57d41641-3d08-45d2-b4a2-124cbe13d0b8-kube-api-access-ww89j\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:49:52.887902 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.887866    2557 generic.go:356] "Generic (PLEG): container finished" podID="57d41641-3d08-45d2-b4a2-124cbe13d0b8" containerID="582b2fe0449ccbc735f21dede096017c33abffc56813475ce6f53ed1654788c8" exitCode=0
Sep 04 23:49:52.888111 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.887913    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"
Sep 04 23:49:52.888111 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.887911    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr" event={"ID":"57d41641-3d08-45d2-b4a2-124cbe13d0b8","Type":"ContainerDied","Data":"582b2fe0449ccbc735f21dede096017c33abffc56813475ce6f53ed1654788c8"}
Sep 04 23:49:52.888111 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.888047    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr" event={"ID":"57d41641-3d08-45d2-b4a2-124cbe13d0b8","Type":"ContainerDied","Data":"f1140a4ee10f9cfae349b70ff4aba579425add2211a4497d2f486d87b0ad34ce"}
Sep 04 23:49:52.888111 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.888071    2557 scope.go:117] "RemoveContainer" containerID="582b2fe0449ccbc735f21dede096017c33abffc56813475ce6f53ed1654788c8"
Sep 04 23:49:52.898254 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.898238    2557 scope.go:117] "RemoveContainer" containerID="a40a619892aa0baa108c302a8814eb71bed1f482d0f07d1ee36b8a6d9e1bbef8"
Sep 04 23:49:52.906604 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.906576    2557 scope.go:117] "RemoveContainer" containerID="582b2fe0449ccbc735f21dede096017c33abffc56813475ce6f53ed1654788c8"
Sep 04 23:49:52.907019 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:49:52.906982    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582b2fe0449ccbc735f21dede096017c33abffc56813475ce6f53ed1654788c8\": container with ID starting with 582b2fe0449ccbc735f21dede096017c33abffc56813475ce6f53ed1654788c8 not found: ID does not exist" containerID="582b2fe0449ccbc735f21dede096017c33abffc56813475ce6f53ed1654788c8"
Sep 04 23:49:52.907129 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.907031    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582b2fe0449ccbc735f21dede096017c33abffc56813475ce6f53ed1654788c8"} err="failed to get container status \"582b2fe0449ccbc735f21dede096017c33abffc56813475ce6f53ed1654788c8\": rpc error: code = NotFound desc = could not find container \"582b2fe0449ccbc735f21dede096017c33abffc56813475ce6f53ed1654788c8\": container with ID starting with 582b2fe0449ccbc735f21dede096017c33abffc56813475ce6f53ed1654788c8 not found: ID does not exist"
Sep 04 23:49:52.907129 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.907055    2557 scope.go:117] "RemoveContainer" containerID="a40a619892aa0baa108c302a8814eb71bed1f482d0f07d1ee36b8a6d9e1bbef8"
Sep 04 23:49:52.907509 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:49:52.907457    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40a619892aa0baa108c302a8814eb71bed1f482d0f07d1ee36b8a6d9e1bbef8\": container with ID starting with a40a619892aa0baa108c302a8814eb71bed1f482d0f07d1ee36b8a6d9e1bbef8 not found: ID does not exist" containerID="a40a619892aa0baa108c302a8814eb71bed1f482d0f07d1ee36b8a6d9e1bbef8"
Sep 04 23:49:52.907509 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.907489    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40a619892aa0baa108c302a8814eb71bed1f482d0f07d1ee36b8a6d9e1bbef8"} err="failed to get container status \"a40a619892aa0baa108c302a8814eb71bed1f482d0f07d1ee36b8a6d9e1bbef8\": rpc error: code = NotFound desc = could not find container \"a40a619892aa0baa108c302a8814eb71bed1f482d0f07d1ee36b8a6d9e1bbef8\": container with ID starting with a40a619892aa0baa108c302a8814eb71bed1f482d0f07d1ee36b8a6d9e1bbef8 not found: ID does not exist"
Sep 04 23:49:52.917674 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.914009    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"]
Sep 04 23:49:52.920584 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:52.920564    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8a8c53fa-gw-openshift-default-5f67bbddbd-kmnpr"]
Sep 04 23:49:54.079550 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:54.079519    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57d41641-3d08-45d2-b4a2-124cbe13d0b8" path="/var/lib/kubelet/pods/57d41641-3d08-45d2-b4a2-124cbe13d0b8/volumes"
Sep 04 23:49:55.421391 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:49:55.421359    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:50:01.883548 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:01.883515    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"
Sep 04 23:50:12.314575 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:12.314544    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:50:13.418510 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:13.418472    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s"]
Sep 04 23:50:13.419028 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:13.418838    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57d41641-3d08-45d2-b4a2-124cbe13d0b8" containerName="istio-proxy"
Sep 04 23:50:13.419028 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:13.418849    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="57d41641-3d08-45d2-b4a2-124cbe13d0b8" containerName="istio-proxy"
Sep 04 23:50:13.419028 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:13.418923    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="57d41641-3d08-45d2-b4a2-124cbe13d0b8" containerName="istio-proxy"
Sep 04 23:50:13.419028 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:13.418931    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="57d41641-3d08-45d2-b4a2-124cbe13d0b8" containerName="istio-proxy"
Sep 04 23:50:13.423622 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:13.423606    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s"
Sep 04 23:50:13.426405 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:13.426378    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 23:50:13.426525 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:13.426378    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k4f64\""
Sep 04 23:50:13.428837 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:13.428814    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s"]
Sep 04 23:50:13.568180 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:13.568148    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb07e711-bd45-4233-8e4b-143ca4c8b035-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s\" (UID: \"bb07e711-bd45-4233-8e4b-143ca4c8b035\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s"
Sep 04 23:50:13.669655 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:13.669580    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb07e711-bd45-4233-8e4b-143ca4c8b035-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s\" (UID: \"bb07e711-bd45-4233-8e4b-143ca4c8b035\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s"
Sep 04 23:50:13.670176 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:13.670158    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb07e711-bd45-4233-8e4b-143ca4c8b035-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s\" (UID: \"bb07e711-bd45-4233-8e4b-143ca4c8b035\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s"
Sep 04 23:50:13.735674 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:13.735652    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s"
Sep 04 23:50:14.002321 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:14.002301    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s"]
Sep 04 23:50:14.008357 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:50:14.008318    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb07e711_bd45_4233_8e4b_143ca4c8b035.slice/crio-df13d764bf4cdec03cc6b432a7d39957c8dd0cf6389e7a0be77d708b8fea1f0e WatchSource:0}: Error finding container df13d764bf4cdec03cc6b432a7d39957c8dd0cf6389e7a0be77d708b8fea1f0e: Status 404 returned error can't find the container with id df13d764bf4cdec03cc6b432a7d39957c8dd0cf6389e7a0be77d708b8fea1f0e
Sep 04 23:50:15.018420 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:15.018391    2557 generic.go:356] "Generic (PLEG): container finished" podID="bb07e711-bd45-4233-8e4b-143ca4c8b035" containerID="775583b87e2bd33cc719e2bb43d2e81a844f652b069fdfd838b2407bcdb22799" exitCode=0
Sep 04 23:50:15.018775 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:15.018475    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s" event={"ID":"bb07e711-bd45-4233-8e4b-143ca4c8b035","Type":"ContainerDied","Data":"775583b87e2bd33cc719e2bb43d2e81a844f652b069fdfd838b2407bcdb22799"}
Sep 04 23:50:15.018775 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:15.018548    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s" event={"ID":"bb07e711-bd45-4233-8e4b-143ca4c8b035","Type":"ContainerStarted","Data":"df13d764bf4cdec03cc6b432a7d39957c8dd0cf6389e7a0be77d708b8fea1f0e"}
Sep 04 23:50:16.150157 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:16.150135    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s"
Sep 04 23:50:16.296668 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:16.296580    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb07e711-bd45-4233-8e4b-143ca4c8b035-maas-api-service-ca\") pod \"bb07e711-bd45-4233-8e4b-143ca4c8b035\" (UID: \"bb07e711-bd45-4233-8e4b-143ca4c8b035\") "
Sep 04 23:50:16.296917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:16.296896    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb07e711-bd45-4233-8e4b-143ca4c8b035-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb07e711-bd45-4233-8e4b-143ca4c8b035" (UID: "bb07e711-bd45-4233-8e4b-143ca4c8b035"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:50:16.398198 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:16.398168    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb07e711-bd45-4233-8e4b-143ca4c8b035-maas-api-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:17.031079 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:17.031047    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s"
Sep 04 23:50:17.031262 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:17.031087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s" event={"ID":"bb07e711-bd45-4233-8e4b-143ca4c8b035","Type":"ContainerDied","Data":"df13d764bf4cdec03cc6b432a7d39957c8dd0cf6389e7a0be77d708b8fea1f0e"}
Sep 04 23:50:17.031262 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:17.031141    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df13d764bf4cdec03cc6b432a7d39957c8dd0cf6389e7a0be77d708b8fea1f0e"
Sep 04 23:50:19.413490 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:19.413456    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"]
Sep 04 23:50:19.413874 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:19.413697    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt" podUID="fd0dc2ad-dd0e-42be-a992-e1b39596d744" containerName="payload-processing" containerID="cri-o://5d4c9104b5858dce03cdb862a35d335764d1806c657d65c4295b6708b755d6d4" gracePeriod=30
Sep 04 23:50:19.561371 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:19.561322    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s"]
Sep 04 23:50:19.564395 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:19.564374    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b52c7b74-gm24s"]
Sep 04 23:50:19.668699 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:19.668646    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"
Sep 04 23:50:19.826787 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:19.826756    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5n8l\" (UniqueName: \"kubernetes.io/projected/fd0dc2ad-dd0e-42be-a992-e1b39596d744-kube-api-access-w5n8l\") pod \"fd0dc2ad-dd0e-42be-a992-e1b39596d744\" (UID: \"fd0dc2ad-dd0e-42be-a992-e1b39596d744\") "
Sep 04 23:50:19.826983 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:19.826848    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0dc2ad-dd0e-42be-a992-e1b39596d744-plugins-config-volume\") pod \"fd0dc2ad-dd0e-42be-a992-e1b39596d744\" (UID: \"fd0dc2ad-dd0e-42be-a992-e1b39596d744\") "
Sep 04 23:50:19.827269 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:19.827248    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd0dc2ad-dd0e-42be-a992-e1b39596d744-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd0dc2ad-dd0e-42be-a992-e1b39596d744" (UID: "fd0dc2ad-dd0e-42be-a992-e1b39596d744"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:50:19.828822 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:19.828804    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0dc2ad-dd0e-42be-a992-e1b39596d744-kube-api-access-w5n8l" (OuterVolumeSpecName: "kube-api-access-w5n8l") pod "fd0dc2ad-dd0e-42be-a992-e1b39596d744" (UID: "fd0dc2ad-dd0e-42be-a992-e1b39596d744"). InnerVolumeSpecName "kube-api-access-w5n8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:50:19.927438 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:19.927372    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5n8l\" (UniqueName: \"kubernetes.io/projected/fd0dc2ad-dd0e-42be-a992-e1b39596d744-kube-api-access-w5n8l\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:19.927438 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:19.927402    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0dc2ad-dd0e-42be-a992-e1b39596d744-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:20.048767 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:20.048734    2557 generic.go:356] "Generic (PLEG): container finished" podID="fd0dc2ad-dd0e-42be-a992-e1b39596d744" containerID="5d4c9104b5858dce03cdb862a35d335764d1806c657d65c4295b6708b755d6d4" exitCode=0
Sep 04 23:50:20.048929 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:20.048785    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"
Sep 04 23:50:20.048929 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:20.048819    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt" event={"ID":"fd0dc2ad-dd0e-42be-a992-e1b39596d744","Type":"ContainerDied","Data":"5d4c9104b5858dce03cdb862a35d335764d1806c657d65c4295b6708b755d6d4"}
Sep 04 23:50:20.048929 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:20.048888    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt" event={"ID":"fd0dc2ad-dd0e-42be-a992-e1b39596d744","Type":"ContainerDied","Data":"db3b8d52cb31bd5ec89e7ac3d71fb2d85fab0ec5a7cf4744f68b1ed4f8152c57"}
Sep 04 23:50:20.048929 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:20.048911    2557 scope.go:117] "RemoveContainer" containerID="5d4c9104b5858dce03cdb862a35d335764d1806c657d65c4295b6708b755d6d4"
Sep 04 23:50:20.058275 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:20.058254    2557 scope.go:117] "RemoveContainer" containerID="5d4c9104b5858dce03cdb862a35d335764d1806c657d65c4295b6708b755d6d4"
Sep 04 23:50:20.058521 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:50:20.058502    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4c9104b5858dce03cdb862a35d335764d1806c657d65c4295b6708b755d6d4\": container with ID starting with 5d4c9104b5858dce03cdb862a35d335764d1806c657d65c4295b6708b755d6d4 not found: ID does not exist" containerID="5d4c9104b5858dce03cdb862a35d335764d1806c657d65c4295b6708b755d6d4"
Sep 04 23:50:20.058594 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:20.058528    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4c9104b5858dce03cdb862a35d335764d1806c657d65c4295b6708b755d6d4"} err="failed to get container status \"5d4c9104b5858dce03cdb862a35d335764d1806c657d65c4295b6708b755d6d4\": rpc error: code = NotFound desc = could not find container \"5d4c9104b5858dce03cdb862a35d335764d1806c657d65c4295b6708b755d6d4\": container with ID starting with 5d4c9104b5858dce03cdb862a35d335764d1806c657d65c4295b6708b755d6d4 not found: ID does not exist"
Sep 04 23:50:20.071796 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:20.071772    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"]
Sep 04 23:50:20.078925 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:20.078896    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb07e711-bd45-4233-8e4b-143ca4c8b035" path="/var/lib/kubelet/pods/bb07e711-bd45-4233-8e4b-143ca4c8b035/volumes"
Sep 04 23:50:20.082012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:20.081991    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b52c7b74-599dcd5db7-bzwnt"]
Sep 04 23:50:21.277859 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.277777    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"]
Sep 04 23:50:21.278351 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.278334    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57d41641-3d08-45d2-b4a2-124cbe13d0b8" containerName="istio-proxy"
Sep 04 23:50:21.278399 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.278354    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="57d41641-3d08-45d2-b4a2-124cbe13d0b8" containerName="istio-proxy"
Sep 04 23:50:21.278399 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.278369    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd0dc2ad-dd0e-42be-a992-e1b39596d744" containerName="payload-processing"
Sep 04 23:50:21.278399 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.278378    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd0dc2ad-dd0e-42be-a992-e1b39596d744" containerName="payload-processing"
Sep 04 23:50:21.278491 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.278405    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb07e711-bd45-4233-8e4b-143ca4c8b035" containerName="revoke-keys"
Sep 04 23:50:21.278491 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.278414    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb07e711-bd45-4233-8e4b-143ca4c8b035" containerName="revoke-keys"
Sep 04 23:50:21.278566 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.278516    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd0dc2ad-dd0e-42be-a992-e1b39596d744" containerName="payload-processing"
Sep 04 23:50:21.278566 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.278531    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb07e711-bd45-4233-8e4b-143ca4c8b035" containerName="revoke-keys"
Sep 04 23:50:21.282997 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.282978    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.287105 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.287086    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-58fce2d8-openshift-default-dockercfg-bzdtq\""
Sep 04 23:50:21.293000 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.292975    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"]
Sep 04 23:50:21.446156 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.446122    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqjc8\" (UniqueName: \"kubernetes.io/projected/08f394bb-2e0b-4101-954f-3d68b27aa216-kube-api-access-qqjc8\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.446336 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.446172    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-credential-socket\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.446336 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.446227    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-podinfo\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.446336 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.446298    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-envoy\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.446336 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.446328    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f394bb-2e0b-4101-954f-3d68b27aa216-istiod-ca-cert\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.446531 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.446366    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-workload-socket\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.446531 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.446386    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-data\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.446531 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.446431    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-workload-certs\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.446531 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.446460    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-token\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.547964 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.547873    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-podinfo\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.547964 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.547911    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-envoy\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.547964 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.547931    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f394bb-2e0b-4101-954f-3d68b27aa216-istiod-ca-cert\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.548198 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.547981    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-workload-socket\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.548198 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.548062    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-data\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.548198 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.548110    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-workload-certs\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.548198 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.548140    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-token\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.548414 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.548205    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqjc8\" (UniqueName: \"kubernetes.io/projected/08f394bb-2e0b-4101-954f-3d68b27aa216-kube-api-access-qqjc8\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.548414 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.548253    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-credential-socket\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.548414 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.548396    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-workload-socket\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.548585 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.548526    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-data\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.548644 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.548586    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-credential-socket\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.548737 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.548720    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-workload-certs\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.548793 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.548767    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f394bb-2e0b-4101-954f-3d68b27aa216-istiod-ca-cert\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.550228 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.550205    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-envoy\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.550431 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.550414    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-podinfo\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.555542 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.555522    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-token\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.555773 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.555757    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqjc8\" (UniqueName: \"kubernetes.io/projected/08f394bb-2e0b-4101-954f-3d68b27aa216-kube-api-access-qqjc8\") pod \"e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") " pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.594899 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.594879    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:21.868963 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.868921    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"]
Sep 04 23:50:21.870159 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:50:21.870136    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f394bb_2e0b_4101_954f_3d68b27aa216.slice/crio-2d154360b6176db940c4c3bd45a186e15a887e216a458a8cf8161391309cbb92 WatchSource:0}: Error finding container 2d154360b6176db940c4c3bd45a186e15a887e216a458a8cf8161391309cbb92: Status 404 returned error can't find the container with id 2d154360b6176db940c4c3bd45a186e15a887e216a458a8cf8161391309cbb92
Sep 04 23:50:21.872441 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.872404    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:50:21.872528 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.872504    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:50:21.872573 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:21.872551    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:50:22.069082 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:22.069046    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz" event={"ID":"08f394bb-2e0b-4101-954f-3d68b27aa216","Type":"ContainerStarted","Data":"b6206011504058ee3eb24b157cfefc3aec9435f23ad772190573ef3ffc80088a"}
Sep 04 23:50:22.069274 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:22.069114    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz" event={"ID":"08f394bb-2e0b-4101-954f-3d68b27aa216","Type":"ContainerStarted","Data":"2d154360b6176db940c4c3bd45a186e15a887e216a458a8cf8161391309cbb92"}
Sep 04 23:50:22.079544 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:22.079520    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd0dc2ad-dd0e-42be-a992-e1b39596d744" path="/var/lib/kubelet/pods/fd0dc2ad-dd0e-42be-a992-e1b39596d744/volumes"
Sep 04 23:50:22.088858 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:22.088819    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz" podStartSLOduration=1.088806871 podStartE2EDuration="1.088806871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:50:22.086453126 +0000 UTC m=+804.504271440" watchObservedRunningTime="2026-09-04 23:50:22.088806871 +0000 UTC m=+804.506625185"
Sep 04 23:50:22.595373 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:22.595340    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:22.600414 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:22.600391    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:23.074984 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:23.074957    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:23.075748 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:23.075733    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:25.377144 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:25.377107    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"]
Sep 04 23:50:25.380567 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:25.380551    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"
Sep 04 23:50:25.383281 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:25.383252    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-58fce2d8\""
Sep 04 23:50:25.383412 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:25.383299    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-58fce2d8-dockercfg-q764j\""
Sep 04 23:50:25.388901 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:25.388881    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"]
Sep 04 23:50:25.486927 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:25.486895    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b100ab5d-4814-4be8-8bae-318e1f20af61-plugins-config-volume\") pod \"payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k\" (UID: \"b100ab5d-4814-4be8-8bae-318e1f20af61\") " pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"
Sep 04 23:50:25.487105 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:25.486957    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqk7d\" (UniqueName: \"kubernetes.io/projected/b100ab5d-4814-4be8-8bae-318e1f20af61-kube-api-access-gqk7d\") pod \"payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k\" (UID: \"b100ab5d-4814-4be8-8bae-318e1f20af61\") " pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"
Sep 04 23:50:25.588382 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:25.588350    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b100ab5d-4814-4be8-8bae-318e1f20af61-plugins-config-volume\") pod \"payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k\" (UID: \"b100ab5d-4814-4be8-8bae-318e1f20af61\") " pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"
Sep 04 23:50:25.588524 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:25.588399    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqk7d\" (UniqueName: \"kubernetes.io/projected/b100ab5d-4814-4be8-8bae-318e1f20af61-kube-api-access-gqk7d\") pod \"payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k\" (UID: \"b100ab5d-4814-4be8-8bae-318e1f20af61\") " pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"
Sep 04 23:50:25.589035 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:25.589016    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b100ab5d-4814-4be8-8bae-318e1f20af61-plugins-config-volume\") pod \"payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k\" (UID: \"b100ab5d-4814-4be8-8bae-318e1f20af61\") " pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"
Sep 04 23:50:25.595631 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:25.595609    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqk7d\" (UniqueName: \"kubernetes.io/projected/b100ab5d-4814-4be8-8bae-318e1f20af61-kube-api-access-gqk7d\") pod \"payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k\" (UID: \"b100ab5d-4814-4be8-8bae-318e1f20af61\") " pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"
Sep 04 23:50:25.693738 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:25.693672    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"
Sep 04 23:50:25.920526 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:25.920494    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:50:25.963515 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:25.963485    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"]
Sep 04 23:50:25.964616 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:50:25.964594    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb100ab5d_4814_4be8_8bae_318e1f20af61.slice/crio-dc0eb34cf2be73fd0abe2a9153fefbcdbed668fd77591da10186c12f2b4ad8ab WatchSource:0}: Error finding container dc0eb34cf2be73fd0abe2a9153fefbcdbed668fd77591da10186c12f2b4ad8ab: Status 404 returned error can't find the container with id dc0eb34cf2be73fd0abe2a9153fefbcdbed668fd77591da10186c12f2b4ad8ab
Sep 04 23:50:26.100609 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:26.100569    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k" event={"ID":"b100ab5d-4814-4be8-8bae-318e1f20af61","Type":"ContainerStarted","Data":"62c9a5fe2bde9b35361fc5068e350ce9fe59055f7f929f154c749e831adf3e77"}
Sep 04 23:50:26.100898 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:26.100626    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k" event={"ID":"b100ab5d-4814-4be8-8bae-318e1f20af61","Type":"ContainerStarted","Data":"dc0eb34cf2be73fd0abe2a9153fefbcdbed668fd77591da10186c12f2b4ad8ab"}
Sep 04 23:50:26.100898 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:26.100658    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"
Sep 04 23:50:26.117414 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:26.117365    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k" podStartSLOduration=1.117346442 podStartE2EDuration="1.117346442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:50:26.11488987 +0000 UTC m=+808.532708191" watchObservedRunningTime="2026-09-04 23:50:26.117346442 +0000 UTC m=+808.535164785"
Sep 04 23:50:37.108680 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:37.108646    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"
Sep 04 23:50:42.121026 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:42.120976    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"]
Sep 04 23:50:43.036054 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:43.036014    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"]
Sep 04 23:50:44.163493 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:44.163450    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9"]
Sep 04 23:50:44.167649 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:44.167628    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9"
Sep 04 23:50:44.172348 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:44.170206    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 23:50:44.172348 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:44.170279    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k4f64\""
Sep 04 23:50:44.172833 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:44.172809    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9"]
Sep 04 23:50:44.248807 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:44.248775    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad43fca-08f7-454f-9405-88eda6f27ebb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-3c6688-bqzj9\" (UID: \"8ad43fca-08f7-454f-9405-88eda6f27ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9"
Sep 04 23:50:44.328349 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:44.328312    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:50:44.350284 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:44.350247    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad43fca-08f7-454f-9405-88eda6f27ebb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-3c6688-bqzj9\" (UID: \"8ad43fca-08f7-454f-9405-88eda6f27ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9"
Sep 04 23:50:44.351038 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:44.351013    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad43fca-08f7-454f-9405-88eda6f27ebb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-3c6688-bqzj9\" (UID: \"8ad43fca-08f7-454f-9405-88eda6f27ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9"
Sep 04 23:50:44.482761 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:44.482724    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9"
Sep 04 23:50:44.756162 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:44.756139    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9"]
Sep 04 23:50:44.757870 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:50:44.757845    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad43fca_08f7_454f_9405_88eda6f27ebb.slice/crio-061e0546493d322042226aa389f3012de92d0476eaf1993e0f4a790dcf60ef79 WatchSource:0}: Error finding container 061e0546493d322042226aa389f3012de92d0476eaf1993e0f4a790dcf60ef79: Status 404 returned error can't find the container with id 061e0546493d322042226aa389f3012de92d0476eaf1993e0f4a790dcf60ef79
Sep 04 23:50:45.233635 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:45.233608    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9" event={"ID":"8ad43fca-08f7-454f-9405-88eda6f27ebb","Type":"ContainerStarted","Data":"c2ea612c12e8e5437295551a6d2074fce8f251011250a2013a84b7f4cf294cba"}
Sep 04 23:50:45.234075 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:45.233653    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9" event={"ID":"8ad43fca-08f7-454f-9405-88eda6f27ebb","Type":"ContainerStarted","Data":"061e0546493d322042226aa389f3012de92d0476eaf1993e0f4a790dcf60ef79"}
Sep 04 23:50:45.249180 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:45.249138    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9" podStartSLOduration=1.249123043 podStartE2EDuration="1.249123043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:50:45.247108955 +0000 UTC m=+827.664927274" watchObservedRunningTime="2026-09-04 23:50:45.249123043 +0000 UTC m=+827.666941362"
Sep 04 23:50:46.248241 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:46.248205    2557 generic.go:356] "Generic (PLEG): container finished" podID="8ad43fca-08f7-454f-9405-88eda6f27ebb" containerID="c2ea612c12e8e5437295551a6d2074fce8f251011250a2013a84b7f4cf294cba" exitCode=0
Sep 04 23:50:46.249457 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:46.248284    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9" event={"ID":"8ad43fca-08f7-454f-9405-88eda6f27ebb","Type":"ContainerDied","Data":"c2ea612c12e8e5437295551a6d2074fce8f251011250a2013a84b7f4cf294cba"}
Sep 04 23:50:47.255820 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:47.255787    2557 generic.go:356] "Generic (PLEG): container finished" podID="08f394bb-2e0b-4101-954f-3d68b27aa216" containerID="b6206011504058ee3eb24b157cfefc3aec9435f23ad772190573ef3ffc80088a" exitCode=0
Sep 04 23:50:47.256313 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:47.255862    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz" event={"ID":"08f394bb-2e0b-4101-954f-3d68b27aa216","Type":"ContainerDied","Data":"b6206011504058ee3eb24b157cfefc3aec9435f23ad772190573ef3ffc80088a"}
Sep 04 23:50:47.256777 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:47.256317    2557 scope.go:117] "RemoveContainer" containerID="b6206011504058ee3eb24b157cfefc3aec9435f23ad772190573ef3ffc80088a"
Sep 04 23:50:47.257416 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:47.257382    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:50:47.257484 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:47.257466    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:50:47.257523 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:47.257498    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:50:47.400307 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:47.400253    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9"
Sep 04 23:50:47.481607 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:47.481579    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad43fca-08f7-454f-9405-88eda6f27ebb-maas-api-service-ca\") pod \"8ad43fca-08f7-454f-9405-88eda6f27ebb\" (UID: \"8ad43fca-08f7-454f-9405-88eda6f27ebb\") "
Sep 04 23:50:47.481983 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:47.481918    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad43fca-08f7-454f-9405-88eda6f27ebb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ad43fca-08f7-454f-9405-88eda6f27ebb" (UID: "8ad43fca-08f7-454f-9405-88eda6f27ebb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:50:47.583234 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:47.583157    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad43fca-08f7-454f-9405-88eda6f27ebb-maas-api-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:48.232977 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.232946    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6"]
Sep 04 23:50:48.233337 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.233325    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ad43fca-08f7-454f-9405-88eda6f27ebb" containerName="revoke-keys"
Sep 04 23:50:48.233389 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.233338    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ad43fca-08f7-454f-9405-88eda6f27ebb" containerName="revoke-keys"
Sep 04 23:50:48.233427 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.233398    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ad43fca-08f7-454f-9405-88eda6f27ebb" containerName="revoke-keys"
Sep 04 23:50:48.235335 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.235320    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6"
Sep 04 23:50:48.243825 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.243800    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6"]
Sep 04 23:50:48.265853 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.265824    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz" event={"ID":"08f394bb-2e0b-4101-954f-3d68b27aa216","Type":"ContainerStarted","Data":"e983d20dfab921f7bcdd819f41d071f75a72a707d57b76f5c8b06c75865c5a99"}
Sep 04 23:50:48.267644 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.267624    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9"
Sep 04 23:50:48.267756 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.267665    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9" event={"ID":"8ad43fca-08f7-454f-9405-88eda6f27ebb","Type":"ContainerDied","Data":"061e0546493d322042226aa389f3012de92d0476eaf1993e0f4a790dcf60ef79"}
Sep 04 23:50:48.267756 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.267711    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="061e0546493d322042226aa389f3012de92d0476eaf1993e0f4a790dcf60ef79"
Sep 04 23:50:48.298961 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.290129    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80be87c4-b5a8-491e-a86d-91722b20bdd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6\" (UID: \"80be87c4-b5a8-491e-a86d-91722b20bdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6"
Sep 04 23:50:48.396631 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.396515    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80be87c4-b5a8-491e-a86d-91722b20bdd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6\" (UID: \"80be87c4-b5a8-491e-a86d-91722b20bdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6"
Sep 04 23:50:48.397400 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.397368    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80be87c4-b5a8-491e-a86d-91722b20bdd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6\" (UID: \"80be87c4-b5a8-491e-a86d-91722b20bdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6"
Sep 04 23:50:48.548004 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.547972    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6"
Sep 04 23:50:48.601422 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.601097    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:48.603319 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.602990    2557 patch_prober.go:28] interesting pod/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 04 23:50:48.603319 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.603050    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz" podUID="08f394bb-2e0b-4101-954f-3d68b27aa216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 04 23:50:48.836537 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:48.836513    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6"]
Sep 04 23:50:48.838386 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:50:48.838360    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80be87c4_b5a8_491e_a86d_91722b20bdd5.slice/crio-35f6b374d9773a0a6659a75b1b99e967f051d87158cc09cae4678a389da9f89f WatchSource:0}: Error finding container 35f6b374d9773a0a6659a75b1b99e967f051d87158cc09cae4678a389da9f89f: Status 404 returned error can't find the container with id 35f6b374d9773a0a6659a75b1b99e967f051d87158cc09cae4678a389da9f89f
Sep 04 23:50:49.275914 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:49.275881    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6" event={"ID":"80be87c4-b5a8-491e-a86d-91722b20bdd5","Type":"ContainerStarted","Data":"f1d7713249a04898a5674f0acc46c820a9c19a7ab002485ba31c44ed5fe37513"}
Sep 04 23:50:49.276333 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:49.275930    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6" event={"ID":"80be87c4-b5a8-491e-a86d-91722b20bdd5","Type":"ContainerStarted","Data":"35f6b374d9773a0a6659a75b1b99e967f051d87158cc09cae4678a389da9f89f"}
Sep 04 23:50:49.293609 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:49.293549    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6" podStartSLOduration=1.293528417 podStartE2EDuration="1.293528417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:50:49.289223624 +0000 UTC m=+831.707041939" watchObservedRunningTime="2026-09-04 23:50:49.293528417 +0000 UTC m=+831.711346737"
Sep 04 23:50:49.596078 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:49.596042    2557 patch_prober.go:28] interesting pod/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 04 23:50:49.596342 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:49.596315    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz" podUID="08f394bb-2e0b-4101-954f-3d68b27aa216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 04 23:50:50.304700 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:50.304649    2557 generic.go:356] "Generic (PLEG): container finished" podID="80be87c4-b5a8-491e-a86d-91722b20bdd5" containerID="f1d7713249a04898a5674f0acc46c820a9c19a7ab002485ba31c44ed5fe37513" exitCode=0
Sep 04 23:50:50.305213 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:50.304998    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6" event={"ID":"80be87c4-b5a8-491e-a86d-91722b20bdd5","Type":"ContainerDied","Data":"f1d7713249a04898a5674f0acc46c820a9c19a7ab002485ba31c44ed5fe37513"}
Sep 04 23:50:50.595326 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:50.595231    2557 patch_prober.go:28] interesting pod/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 04 23:50:50.595326 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:50.595296    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz" podUID="08f394bb-2e0b-4101-954f-3d68b27aa216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 04 23:50:51.459853 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:51.459832    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6"
Sep 04 23:50:51.527614 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:51.527583    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80be87c4-b5a8-491e-a86d-91722b20bdd5-maas-api-service-ca\") pod \"80be87c4-b5a8-491e-a86d-91722b20bdd5\" (UID: \"80be87c4-b5a8-491e-a86d-91722b20bdd5\") "
Sep 04 23:50:51.527916 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:51.527893    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80be87c4-b5a8-491e-a86d-91722b20bdd5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80be87c4-b5a8-491e-a86d-91722b20bdd5" (UID: "80be87c4-b5a8-491e-a86d-91722b20bdd5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:50:51.599009 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:51.598282    2557 patch_prober.go:28] interesting pod/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 04 23:50:51.599009 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:51.598344    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz" podUID="08f394bb-2e0b-4101-954f-3d68b27aa216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 04 23:50:51.629152 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:51.629125    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80be87c4-b5a8-491e-a86d-91722b20bdd5-maas-api-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:52.318383 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:52.318351    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6"
Sep 04 23:50:52.318561 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:52.318389    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6" event={"ID":"80be87c4-b5a8-491e-a86d-91722b20bdd5","Type":"ContainerDied","Data":"35f6b374d9773a0a6659a75b1b99e967f051d87158cc09cae4678a389da9f89f"}
Sep 04 23:50:52.318561 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:52.318451    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35f6b374d9773a0a6659a75b1b99e967f051d87158cc09cae4678a389da9f89f"
Sep 04 23:50:52.595779 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:52.595745    2557 patch_prober.go:28] interesting pod/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 04 23:50:52.596354 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:52.596325    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz" podUID="08f394bb-2e0b-4101-954f-3d68b27aa216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 04 23:50:53.074870 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:53.074840    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:53.596137 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:53.596100    2557 patch_prober.go:28] interesting pod/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 04 23:50:53.596645 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:53.596152    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz" podUID="08f394bb-2e0b-4101-954f-3d68b27aa216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 04 23:50:54.600265 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:54.600236    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:54.601276 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:54.601253    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:50:54.927377 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:54.927291    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"]
Sep 04 23:50:54.927580 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:54.927552    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k" podUID="b100ab5d-4814-4be8-8bae-318e1f20af61" containerName="payload-processing" containerID="cri-o://62c9a5fe2bde9b35361fc5068e350ce9fe59055f7f929f154c749e831adf3e77" gracePeriod=30
Sep 04 23:50:55.072139 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.072103    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"]
Sep 04 23:50:55.072634 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.072597    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv" podUID="96f278e3-dc59-4c45-84d2-418f21dd7082" containerName="payload-pre-processing" containerID="cri-o://1fa0f41e8d04bb94606215ff402d397cec04fdaef8f56ac5a2c7f642e4bccbca" gracePeriod=30
Sep 04 23:50:55.103253 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.102913    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6"]
Sep 04 23:50:55.107047 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.106959    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58fce2d8-7xjp6"]
Sep 04 23:50:55.225059 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.225032    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"]
Sep 04 23:50:55.234800 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.234773    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"
Sep 04 23:50:55.325093 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.325063    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9"]
Sep 04 23:50:55.327544 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.327524    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3c6688-bqzj9"]
Sep 04 23:50:55.343021 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.342993    2557 generic.go:356] "Generic (PLEG): container finished" podID="b100ab5d-4814-4be8-8bae-318e1f20af61" containerID="62c9a5fe2bde9b35361fc5068e350ce9fe59055f7f929f154c749e831adf3e77" exitCode=0
Sep 04 23:50:55.343154 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.343043    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"
Sep 04 23:50:55.343154 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.343073    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k" event={"ID":"b100ab5d-4814-4be8-8bae-318e1f20af61","Type":"ContainerDied","Data":"62c9a5fe2bde9b35361fc5068e350ce9fe59055f7f929f154c749e831adf3e77"}
Sep 04 23:50:55.343253 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.343152    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k" event={"ID":"b100ab5d-4814-4be8-8bae-318e1f20af61","Type":"ContainerDied","Data":"dc0eb34cf2be73fd0abe2a9153fefbcdbed668fd77591da10186c12f2b4ad8ab"}
Sep 04 23:50:55.343253 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.343182    2557 scope.go:117] "RemoveContainer" containerID="62c9a5fe2bde9b35361fc5068e350ce9fe59055f7f929f154c749e831adf3e77"
Sep 04 23:50:55.344856 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.344829    2557 generic.go:356] "Generic (PLEG): container finished" podID="96f278e3-dc59-4c45-84d2-418f21dd7082" containerID="1fa0f41e8d04bb94606215ff402d397cec04fdaef8f56ac5a2c7f642e4bccbca" exitCode=0
Sep 04 23:50:55.345045 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.344907    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv" event={"ID":"96f278e3-dc59-4c45-84d2-418f21dd7082","Type":"ContainerDied","Data":"1fa0f41e8d04bb94606215ff402d397cec04fdaef8f56ac5a2c7f642e4bccbca"}
Sep 04 23:50:55.345258 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.345236    2557 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz" secret="" err="secret \"e2e-derive-58fce2d8-openshift-default-dockercfg-bzdtq\" not found"
Sep 04 23:50:55.359717 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.359691    2557 scope.go:117] "RemoveContainer" containerID="62c9a5fe2bde9b35361fc5068e350ce9fe59055f7f929f154c749e831adf3e77"
Sep 04 23:50:55.359966 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:50:55.359931    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c9a5fe2bde9b35361fc5068e350ce9fe59055f7f929f154c749e831adf3e77\": container with ID starting with 62c9a5fe2bde9b35361fc5068e350ce9fe59055f7f929f154c749e831adf3e77 not found: ID does not exist" containerID="62c9a5fe2bde9b35361fc5068e350ce9fe59055f7f929f154c749e831adf3e77"
Sep 04 23:50:55.360038 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.359975    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c9a5fe2bde9b35361fc5068e350ce9fe59055f7f929f154c749e831adf3e77"} err="failed to get container status \"62c9a5fe2bde9b35361fc5068e350ce9fe59055f7f929f154c749e831adf3e77\": rpc error: code = NotFound desc = could not find container \"62c9a5fe2bde9b35361fc5068e350ce9fe59055f7f929f154c749e831adf3e77\": container with ID starting with 62c9a5fe2bde9b35361fc5068e350ce9fe59055f7f929f154c749e831adf3e77 not found: ID does not exist"
Sep 04 23:50:55.361416 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.361399    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"
Sep 04 23:50:55.364695 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.364674    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b100ab5d-4814-4be8-8bae-318e1f20af61-plugins-config-volume\") pod \"b100ab5d-4814-4be8-8bae-318e1f20af61\" (UID: \"b100ab5d-4814-4be8-8bae-318e1f20af61\") "
Sep 04 23:50:55.364778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.364716    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqk7d\" (UniqueName: \"kubernetes.io/projected/b100ab5d-4814-4be8-8bae-318e1f20af61-kube-api-access-gqk7d\") pod \"b100ab5d-4814-4be8-8bae-318e1f20af61\" (UID: \"b100ab5d-4814-4be8-8bae-318e1f20af61\") "
Sep 04 23:50:55.365140 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.365111    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b100ab5d-4814-4be8-8bae-318e1f20af61-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b100ab5d-4814-4be8-8bae-318e1f20af61" (UID: "b100ab5d-4814-4be8-8bae-318e1f20af61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:50:55.366699 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.366681    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b100ab5d-4814-4be8-8bae-318e1f20af61-kube-api-access-gqk7d" (OuterVolumeSpecName: "kube-api-access-gqk7d") pod "b100ab5d-4814-4be8-8bae-318e1f20af61" (UID: "b100ab5d-4814-4be8-8bae-318e1f20af61"). InnerVolumeSpecName "kube-api-access-gqk7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:50:55.466419 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.466391    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f278e3-dc59-4c45-84d2-418f21dd7082-plugins-config-volume\") pod \"96f278e3-dc59-4c45-84d2-418f21dd7082\" (UID: \"96f278e3-dc59-4c45-84d2-418f21dd7082\") "
Sep 04 23:50:55.466588 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.466434    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mftb9\" (UniqueName: \"kubernetes.io/projected/96f278e3-dc59-4c45-84d2-418f21dd7082-kube-api-access-mftb9\") pod \"96f278e3-dc59-4c45-84d2-418f21dd7082\" (UID: \"96f278e3-dc59-4c45-84d2-418f21dd7082\") "
Sep 04 23:50:55.466830 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.466801    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b100ab5d-4814-4be8-8bae-318e1f20af61-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:55.466830 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.466833    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqk7d\" (UniqueName: \"kubernetes.io/projected/b100ab5d-4814-4be8-8bae-318e1f20af61-kube-api-access-gqk7d\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:55.467038 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.466802    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f278e3-dc59-4c45-84d2-418f21dd7082-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96f278e3-dc59-4c45-84d2-418f21dd7082" (UID: "96f278e3-dc59-4c45-84d2-418f21dd7082"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:50:55.468488 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.468467    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f278e3-dc59-4c45-84d2-418f21dd7082-kube-api-access-mftb9" (OuterVolumeSpecName: "kube-api-access-mftb9") pod "96f278e3-dc59-4c45-84d2-418f21dd7082" (UID: "96f278e3-dc59-4c45-84d2-418f21dd7082"). InnerVolumeSpecName "kube-api-access-mftb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:50:55.568069 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.568045    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f278e3-dc59-4c45-84d2-418f21dd7082-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:55.568069 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.568067    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mftb9\" (UniqueName: \"kubernetes.io/projected/96f278e3-dc59-4c45-84d2-418f21dd7082-kube-api-access-mftb9\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:55.663894 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.663869    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"]
Sep 04 23:50:55.667090 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:55.667068    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-58fce2d8-54dcd8dc57-bqm8k"]
Sep 04 23:50:56.079298 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:56.079262    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80be87c4-b5a8-491e-a86d-91722b20bdd5" path="/var/lib/kubelet/pods/80be87c4-b5a8-491e-a86d-91722b20bdd5/volumes"
Sep 04 23:50:56.079656 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:56.079641    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ad43fca-08f7-454f-9405-88eda6f27ebb" path="/var/lib/kubelet/pods/8ad43fca-08f7-454f-9405-88eda6f27ebb/volumes"
Sep 04 23:50:56.080012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:56.079998    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b100ab5d-4814-4be8-8bae-318e1f20af61" path="/var/lib/kubelet/pods/b100ab5d-4814-4be8-8bae-318e1f20af61/volumes"
Sep 04 23:50:56.352950 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:56.352911    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"
Sep 04 23:50:56.353164 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:56.352965    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv" event={"ID":"96f278e3-dc59-4c45-84d2-418f21dd7082","Type":"ContainerDied","Data":"cabe2f4b743f570bfc39870b94daf049fa3c27fc2a9093b46fb66240d1155a6f"}
Sep 04 23:50:56.353164 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:56.353041    2557 scope.go:117] "RemoveContainer" containerID="1fa0f41e8d04bb94606215ff402d397cec04fdaef8f56ac5a2c7f642e4bccbca"
Sep 04 23:50:56.354708 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:56.354676    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz" podUID="08f394bb-2e0b-4101-954f-3d68b27aa216" containerName="istio-proxy" containerID="cri-o://e983d20dfab921f7bcdd819f41d071f75a72a707d57b76f5c8b06c75865c5a99" gracePeriod=30
Sep 04 23:50:56.372162 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:56.372134    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"]
Sep 04 23:50:56.375927 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:56.375902    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3c6688-845684cff4-562qv"]
Sep 04 23:50:57.144618 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.144573    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k" podUID="075e7b47-fddd-41f1-a843-b72f05179f51" containerName="main" containerID="cri-o://a64adda5ab83cf193f9ac3ed44452a85308ce27f3e7353d613b834cfc43f5dff" gracePeriod=45
Sep 04 23:50:57.362874 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.362844    2557 generic.go:356] "Generic (PLEG): container finished" podID="075e7b47-fddd-41f1-a843-b72f05179f51" containerID="a64adda5ab83cf193f9ac3ed44452a85308ce27f3e7353d613b834cfc43f5dff" exitCode=0
Sep 04 23:50:57.363049 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.362987    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k" event={"ID":"075e7b47-fddd-41f1-a843-b72f05179f51","Type":"ContainerDied","Data":"a64adda5ab83cf193f9ac3ed44452a85308ce27f3e7353d613b834cfc43f5dff"}
Sep 04 23:50:57.389589 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.389564    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:50:57.488634 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.488605    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wd9z\" (UniqueName: \"kubernetes.io/projected/075e7b47-fddd-41f1-a843-b72f05179f51-kube-api-access-9wd9z\") pod \"075e7b47-fddd-41f1-a843-b72f05179f51\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") "
Sep 04 23:50:57.488783 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.488665    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-model-cache\") pod \"075e7b47-fddd-41f1-a843-b72f05179f51\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") "
Sep 04 23:50:57.488783 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.488705    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-home\") pod \"075e7b47-fddd-41f1-a843-b72f05179f51\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") "
Sep 04 23:50:57.488783 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.488739    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/075e7b47-fddd-41f1-a843-b72f05179f51-tls-certs\") pod \"075e7b47-fddd-41f1-a843-b72f05179f51\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") "
Sep 04 23:50:57.488931 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.488785    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-dshm\") pod \"075e7b47-fddd-41f1-a843-b72f05179f51\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") "
Sep 04 23:50:57.488931 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.488809    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-tmp-dir\") pod \"075e7b47-fddd-41f1-a843-b72f05179f51\" (UID: \"075e7b47-fddd-41f1-a843-b72f05179f51\") "
Sep 04 23:50:57.489068 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.488923    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-model-cache" (OuterVolumeSpecName: "model-cache") pod "075e7b47-fddd-41f1-a843-b72f05179f51" (UID: "075e7b47-fddd-41f1-a843-b72f05179f51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:50:57.489068 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.488990    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-home" (OuterVolumeSpecName: "home") pod "075e7b47-fddd-41f1-a843-b72f05179f51" (UID: "075e7b47-fddd-41f1-a843-b72f05179f51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:50:57.489178 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.489131    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "075e7b47-fddd-41f1-a843-b72f05179f51" (UID: "075e7b47-fddd-41f1-a843-b72f05179f51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:50:57.489226 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.489177    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:57.489226 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.489194    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:57.490839 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.490817    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-dshm" (OuterVolumeSpecName: "dshm") pod "075e7b47-fddd-41f1-a843-b72f05179f51" (UID: "075e7b47-fddd-41f1-a843-b72f05179f51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:50:57.490912 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.490845    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/075e7b47-fddd-41f1-a843-b72f05179f51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "075e7b47-fddd-41f1-a843-b72f05179f51" (UID: "075e7b47-fddd-41f1-a843-b72f05179f51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:50:57.490912 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.490889    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075e7b47-fddd-41f1-a843-b72f05179f51-kube-api-access-9wd9z" (OuterVolumeSpecName: "kube-api-access-9wd9z") pod "075e7b47-fddd-41f1-a843-b72f05179f51" (UID: "075e7b47-fddd-41f1-a843-b72f05179f51"). InnerVolumeSpecName "kube-api-access-9wd9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:50:57.590181 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.590160    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/075e7b47-fddd-41f1-a843-b72f05179f51-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:57.590181 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.590179    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:57.590314 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.590188    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/075e7b47-fddd-41f1-a843-b72f05179f51-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:57.590314 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:57.590196    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wd9z\" (UniqueName: \"kubernetes.io/projected/075e7b47-fddd-41f1-a843-b72f05179f51-kube-api-access-9wd9z\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:58.059351 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.059175    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d" podUID="5b34a041-67c5-4b1f-865b-266e3e727d7d" containerName="main" containerID="cri-o://8cf89cb20c1265e7d08f343c08e7c1decee897eac7d023a6dc71905580a0b691" gracePeriod=45
Sep 04 23:50:58.079587 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.079559    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96f278e3-dc59-4c45-84d2-418f21dd7082" path="/var/lib/kubelet/pods/96f278e3-dc59-4c45-84d2-418f21dd7082/volumes"
Sep 04 23:50:58.298399 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.298372    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:50:58.375739 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.375663    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"
Sep 04 23:50:58.375739 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.375700    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k" event={"ID":"075e7b47-fddd-41f1-a843-b72f05179f51","Type":"ContainerDied","Data":"4dad1e34452b096f723e82bd839cd2bf236ebb019c100637bec1fd0d3fdf8933"}
Sep 04 23:50:58.375960 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.375755    2557 scope.go:117] "RemoveContainer" containerID="a64adda5ab83cf193f9ac3ed44452a85308ce27f3e7353d613b834cfc43f5dff"
Sep 04 23:50:58.377520 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.377481    2557 generic.go:356] "Generic (PLEG): container finished" podID="5b34a041-67c5-4b1f-865b-266e3e727d7d" containerID="8cf89cb20c1265e7d08f343c08e7c1decee897eac7d023a6dc71905580a0b691" exitCode=0
Sep 04 23:50:58.377690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.377621    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d" event={"ID":"5b34a041-67c5-4b1f-865b-266e3e727d7d","Type":"ContainerDied","Data":"8cf89cb20c1265e7d08f343c08e7c1decee897eac7d023a6dc71905580a0b691"}
Sep 04 23:50:58.377690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.377669    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d" event={"ID":"5b34a041-67c5-4b1f-865b-266e3e727d7d","Type":"ContainerDied","Data":"a8daa7f56fd137a6b3ff19851f2291337222c213718a321bbab2ea2f4e41e0e9"}
Sep 04 23:50:58.377838 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.377765    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"
Sep 04 23:50:58.384296 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.384280    2557 scope.go:117] "RemoveContainer" containerID="8cf89cb20c1265e7d08f343c08e7c1decee897eac7d023a6dc71905580a0b691"
Sep 04 23:50:58.392076 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.392062    2557 scope.go:117] "RemoveContainer" containerID="8cf89cb20c1265e7d08f343c08e7c1decee897eac7d023a6dc71905580a0b691"
Sep 04 23:50:58.392325 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:50:58.392304    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf89cb20c1265e7d08f343c08e7c1decee897eac7d023a6dc71905580a0b691\": container with ID starting with 8cf89cb20c1265e7d08f343c08e7c1decee897eac7d023a6dc71905580a0b691 not found: ID does not exist" containerID="8cf89cb20c1265e7d08f343c08e7c1decee897eac7d023a6dc71905580a0b691"
Sep 04 23:50:58.392435 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.392329    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf89cb20c1265e7d08f343c08e7c1decee897eac7d023a6dc71905580a0b691"} err="failed to get container status \"8cf89cb20c1265e7d08f343c08e7c1decee897eac7d023a6dc71905580a0b691\": rpc error: code = NotFound desc = could not find container \"8cf89cb20c1265e7d08f343c08e7c1decee897eac7d023a6dc71905580a0b691\": container with ID starting with 8cf89cb20c1265e7d08f343c08e7c1decee897eac7d023a6dc71905580a0b691 not found: ID does not exist"
Sep 04 23:50:58.394852 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.394832    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b34a041-67c5-4b1f-865b-266e3e727d7d-tls-certs\") pod \"5b34a041-67c5-4b1f-865b-266e3e727d7d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") "
Sep 04 23:50:58.394996 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.394863    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-dshm\") pod \"5b34a041-67c5-4b1f-865b-266e3e727d7d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") "
Sep 04 23:50:58.394996 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.394898    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-model-cache\") pod \"5b34a041-67c5-4b1f-865b-266e3e727d7d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") "
Sep 04 23:50:58.394996 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.394916    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-home\") pod \"5b34a041-67c5-4b1f-865b-266e3e727d7d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") "
Sep 04 23:50:58.394996 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.394986    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-tmp-dir\") pod \"5b34a041-67c5-4b1f-865b-266e3e727d7d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") "
Sep 04 23:50:58.395209 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.395019    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4tjm\" (UniqueName: \"kubernetes.io/projected/5b34a041-67c5-4b1f-865b-266e3e727d7d-kube-api-access-r4tjm\") pod \"5b34a041-67c5-4b1f-865b-266e3e727d7d\" (UID: \"5b34a041-67c5-4b1f-865b-266e3e727d7d\") "
Sep 04 23:50:58.395209 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.395165    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b34a041-67c5-4b1f-865b-266e3e727d7d" (UID: "5b34a041-67c5-4b1f-865b-266e3e727d7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:50:58.395320 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.395260    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-home" (OuterVolumeSpecName: "home") pod "5b34a041-67c5-4b1f-865b-266e3e727d7d" (UID: "5b34a041-67c5-4b1f-865b-266e3e727d7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:50:58.395320 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.395283    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b34a041-67c5-4b1f-865b-266e3e727d7d" (UID: "5b34a041-67c5-4b1f-865b-266e3e727d7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:50:58.395419 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.395347    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:58.395574 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.395552    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"]
Sep 04 23:50:58.397233 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.397211    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-dshm" (OuterVolumeSpecName: "dshm") pod "5b34a041-67c5-4b1f-865b-266e3e727d7d" (UID: "5b34a041-67c5-4b1f-865b-266e3e727d7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:50:58.397355 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.397339    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b34a041-67c5-4b1f-865b-266e3e727d7d-kube-api-access-r4tjm" (OuterVolumeSpecName: "kube-api-access-r4tjm") pod "5b34a041-67c5-4b1f-865b-266e3e727d7d" (UID: "5b34a041-67c5-4b1f-865b-266e3e727d7d"). InnerVolumeSpecName "kube-api-access-r4tjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:50:58.397503 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.397485    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b34a041-67c5-4b1f-865b-266e3e727d7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b34a041-67c5-4b1f-865b-266e3e727d7d" (UID: "5b34a041-67c5-4b1f-865b-266e3e727d7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:50:58.399142 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.399122    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-98e5323e-kserve-65f5cf79c9-zpq2k"]
Sep 04 23:50:58.496644 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.496622    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4tjm\" (UniqueName: \"kubernetes.io/projected/5b34a041-67c5-4b1f-865b-266e3e727d7d-kube-api-access-r4tjm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:58.496644 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.496643    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b34a041-67c5-4b1f-865b-266e3e727d7d-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:58.496773 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.496653    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:58.496773 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.496661    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:58.496773 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.496668    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b34a041-67c5-4b1f-865b-266e3e727d7d-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:50:58.701273 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.701237    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"]
Sep 04 23:50:58.703334 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:50:58.703314    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-98e5323e-kserve-6474d888d9-mrf4d"]
Sep 04 23:51:00.079822 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:00.079791    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="075e7b47-fddd-41f1-a843-b72f05179f51" path="/var/lib/kubelet/pods/075e7b47-fddd-41f1-a843-b72f05179f51/volumes"
Sep 04 23:51:00.080204 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:00.080189    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b34a041-67c5-4b1f-865b-266e3e727d7d" path="/var/lib/kubelet/pods/5b34a041-67c5-4b1f-865b-266e3e727d7d/volumes"
Sep 04 23:51:01.603197 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.603168    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:51:01.723218 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723179    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-podinfo\") pod \"08f394bb-2e0b-4101-954f-3d68b27aa216\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") "
Sep 04 23:51:01.723218 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723215    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f394bb-2e0b-4101-954f-3d68b27aa216-istiod-ca-cert\") pod \"08f394bb-2e0b-4101-954f-3d68b27aa216\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") "
Sep 04 23:51:01.723476 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723244    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-data\") pod \"08f394bb-2e0b-4101-954f-3d68b27aa216\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") "
Sep 04 23:51:01.723476 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723282    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqjc8\" (UniqueName: \"kubernetes.io/projected/08f394bb-2e0b-4101-954f-3d68b27aa216-kube-api-access-qqjc8\") pod \"08f394bb-2e0b-4101-954f-3d68b27aa216\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") "
Sep 04 23:51:01.723476 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723323    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-credential-socket\") pod \"08f394bb-2e0b-4101-954f-3d68b27aa216\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") "
Sep 04 23:51:01.723476 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723359    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-envoy\") pod \"08f394bb-2e0b-4101-954f-3d68b27aa216\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") "
Sep 04 23:51:01.723476 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723384    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-workload-socket\") pod \"08f394bb-2e0b-4101-954f-3d68b27aa216\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") "
Sep 04 23:51:01.723476 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723437    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-workload-certs\") pod \"08f394bb-2e0b-4101-954f-3d68b27aa216\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") "
Sep 04 23:51:01.723476 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723458    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-token\") pod \"08f394bb-2e0b-4101-954f-3d68b27aa216\" (UID: \"08f394bb-2e0b-4101-954f-3d68b27aa216\") "
Sep 04 23:51:01.723831 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723628    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-data" (OuterVolumeSpecName: "istio-data") pod "08f394bb-2e0b-4101-954f-3d68b27aa216" (UID: "08f394bb-2e0b-4101-954f-3d68b27aa216"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:51:01.723831 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723664    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08f394bb-2e0b-4101-954f-3d68b27aa216" (UID: "08f394bb-2e0b-4101-954f-3d68b27aa216"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:51:01.723831 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723690    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f394bb-2e0b-4101-954f-3d68b27aa216-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08f394bb-2e0b-4101-954f-3d68b27aa216" (UID: "08f394bb-2e0b-4101-954f-3d68b27aa216"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:51:01.724012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723836    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08f394bb-2e0b-4101-954f-3d68b27aa216" (UID: "08f394bb-2e0b-4101-954f-3d68b27aa216"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:51:01.724012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723955    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f394bb-2e0b-4101-954f-3d68b27aa216-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:51:01.724012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723979    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:51:01.724012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.723993    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:51:01.724012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.724008    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:51:01.724279 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.724220    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08f394bb-2e0b-4101-954f-3d68b27aa216" (UID: "08f394bb-2e0b-4101-954f-3d68b27aa216"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:51:01.726672 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.726637    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08f394bb-2e0b-4101-954f-3d68b27aa216" (UID: "08f394bb-2e0b-4101-954f-3d68b27aa216"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:51:01.726776 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.726711    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08f394bb-2e0b-4101-954f-3d68b27aa216" (UID: "08f394bb-2e0b-4101-954f-3d68b27aa216"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:51:01.726776 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.726734    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f394bb-2e0b-4101-954f-3d68b27aa216-kube-api-access-qqjc8" (OuterVolumeSpecName: "kube-api-access-qqjc8") pod "08f394bb-2e0b-4101-954f-3d68b27aa216" (UID: "08f394bb-2e0b-4101-954f-3d68b27aa216"). InnerVolumeSpecName "kube-api-access-qqjc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:51:01.726855 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.726809    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-token" (OuterVolumeSpecName: "istio-token") pod "08f394bb-2e0b-4101-954f-3d68b27aa216" (UID: "08f394bb-2e0b-4101-954f-3d68b27aa216"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:51:01.824462 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.824426    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:51:01.824462 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.824453    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f394bb-2e0b-4101-954f-3d68b27aa216-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:51:01.824462 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.824462    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:51:01.824462 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.824469    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f394bb-2e0b-4101-954f-3d68b27aa216-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:51:01.824777 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:01.824478    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqjc8\" (UniqueName: \"kubernetes.io/projected/08f394bb-2e0b-4101-954f-3d68b27aa216-kube-api-access-qqjc8\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:51:02.226221 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.226185    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:51:02.403966 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.403860    2557 generic.go:356] "Generic (PLEG): container finished" podID="08f394bb-2e0b-4101-954f-3d68b27aa216" containerID="e983d20dfab921f7bcdd819f41d071f75a72a707d57b76f5c8b06c75865c5a99" exitCode=0
Sep 04 23:51:02.403966 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.403911    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz" event={"ID":"08f394bb-2e0b-4101-954f-3d68b27aa216","Type":"ContainerDied","Data":"e983d20dfab921f7bcdd819f41d071f75a72a707d57b76f5c8b06c75865c5a99"}
Sep 04 23:51:02.403966 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.403950    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"
Sep 04 23:51:02.404261 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.403983    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz" event={"ID":"08f394bb-2e0b-4101-954f-3d68b27aa216","Type":"ContainerDied","Data":"2d154360b6176db940c4c3bd45a186e15a887e216a458a8cf8161391309cbb92"}
Sep 04 23:51:02.404261 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.404007    2557 scope.go:117] "RemoveContainer" containerID="e983d20dfab921f7bcdd819f41d071f75a72a707d57b76f5c8b06c75865c5a99"
Sep 04 23:51:02.421437 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.421321    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"]
Sep 04 23:51:02.421754 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.421723    2557 scope.go:117] "RemoveContainer" containerID="b6206011504058ee3eb24b157cfefc3aec9435f23ad772190573ef3ffc80088a"
Sep 04 23:51:02.422218 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.422202    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="075e7b47-fddd-41f1-a843-b72f05179f51" containerName="main"
Sep 04 23:51:02.422367 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.422355    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="075e7b47-fddd-41f1-a843-b72f05179f51" containerName="main"
Sep 04 23:51:02.422482 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.422470    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96f278e3-dc59-4c45-84d2-418f21dd7082" containerName="payload-pre-processing"
Sep 04 23:51:02.422576 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.422566    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="96f278e3-dc59-4c45-84d2-418f21dd7082" containerName="payload-pre-processing"
Sep 04 23:51:02.422682 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.422671    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08f394bb-2e0b-4101-954f-3d68b27aa216" containerName="istio-proxy"
Sep 04 23:51:02.422771 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.422760    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="08f394bb-2e0b-4101-954f-3d68b27aa216" containerName="istio-proxy"
Sep 04 23:51:02.422869 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.422859    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80be87c4-b5a8-491e-a86d-91722b20bdd5" containerName="revoke-keys"
Sep 04 23:51:02.422959 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.422950    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="80be87c4-b5a8-491e-a86d-91722b20bdd5" containerName="revoke-keys"
Sep 04 23:51:02.423075 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.423065    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b100ab5d-4814-4be8-8bae-318e1f20af61" containerName="payload-processing"
Sep 04 23:51:02.423167 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.423157    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b100ab5d-4814-4be8-8bae-318e1f20af61" containerName="payload-processing"
Sep 04 23:51:02.423290 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.423280    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b34a041-67c5-4b1f-865b-266e3e727d7d" containerName="main"
Sep 04 23:51:02.423388 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.423377    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b34a041-67c5-4b1f-865b-266e3e727d7d" containerName="main"
Sep 04 23:51:02.424166 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.423882    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="075e7b47-fddd-41f1-a843-b72f05179f51" containerName="main"
Sep 04 23:51:02.424241 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.424173    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b34a041-67c5-4b1f-865b-266e3e727d7d" containerName="main"
Sep 04 23:51:02.424241 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.424184    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b100ab5d-4814-4be8-8bae-318e1f20af61" containerName="payload-processing"
Sep 04 23:51:02.424241 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.424200    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="08f394bb-2e0b-4101-954f-3d68b27aa216" containerName="istio-proxy"
Sep 04 23:51:02.424241 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.424212    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="80be87c4-b5a8-491e-a86d-91722b20bdd5" containerName="revoke-keys"
Sep 04 23:51:02.424241 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.424223    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="96f278e3-dc59-4c45-84d2-418f21dd7082" containerName="payload-pre-processing"
Sep 04 23:51:02.428874 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.428854    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.429857 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.429837    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"]
Sep 04 23:51:02.431241 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.431223    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-6fa8d6-openshift-default-dockercfg-dbhgz\""
Sep 04 23:51:02.433009 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.432985    2557 scope.go:117] "RemoveContainer" containerID="e983d20dfab921f7bcdd819f41d071f75a72a707d57b76f5c8b06c75865c5a99"
Sep 04 23:51:02.433371 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:51:02.433343    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e983d20dfab921f7bcdd819f41d071f75a72a707d57b76f5c8b06c75865c5a99\": container with ID starting with e983d20dfab921f7bcdd819f41d071f75a72a707d57b76f5c8b06c75865c5a99 not found: ID does not exist" containerID="e983d20dfab921f7bcdd819f41d071f75a72a707d57b76f5c8b06c75865c5a99"
Sep 04 23:51:02.433573 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.433515    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e983d20dfab921f7bcdd819f41d071f75a72a707d57b76f5c8b06c75865c5a99"} err="failed to get container status \"e983d20dfab921f7bcdd819f41d071f75a72a707d57b76f5c8b06c75865c5a99\": rpc error: code = NotFound desc = could not find container \"e983d20dfab921f7bcdd819f41d071f75a72a707d57b76f5c8b06c75865c5a99\": container with ID starting with e983d20dfab921f7bcdd819f41d071f75a72a707d57b76f5c8b06c75865c5a99 not found: ID does not exist"
Sep 04 23:51:02.433750 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.433734    2557 scope.go:117] "RemoveContainer" containerID="b6206011504058ee3eb24b157cfefc3aec9435f23ad772190573ef3ffc80088a"
Sep 04 23:51:02.435381 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:51:02.435323    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6206011504058ee3eb24b157cfefc3aec9435f23ad772190573ef3ffc80088a\": container with ID starting with b6206011504058ee3eb24b157cfefc3aec9435f23ad772190573ef3ffc80088a not found: ID does not exist" containerID="b6206011504058ee3eb24b157cfefc3aec9435f23ad772190573ef3ffc80088a"
Sep 04 23:51:02.435569 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.435510    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6206011504058ee3eb24b157cfefc3aec9435f23ad772190573ef3ffc80088a"} err="failed to get container status \"b6206011504058ee3eb24b157cfefc3aec9435f23ad772190573ef3ffc80088a\": rpc error: code = NotFound desc = could not find container \"b6206011504058ee3eb24b157cfefc3aec9435f23ad772190573ef3ffc80088a\": container with ID starting with b6206011504058ee3eb24b157cfefc3aec9435f23ad772190573ef3ffc80088a not found: ID does not exist"
Sep 04 23:51:02.439646 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.438706    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-58fce2d8-openshift-default-7d4f48894d-xx6pz"]
Sep 04 23:51:02.441691 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.441667    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"]
Sep 04 23:51:02.530545 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.530513    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndcdq\" (UniqueName: \"kubernetes.io/projected/3177cae4-d346-4fae-9310-8e5da395bb9e-kube-api-access-ndcdq\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.530670 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.530569    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-workload-certs\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.530670 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.530615    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-credential-socket\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.530670 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.530634    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-token\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.530784 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.530677    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3177cae4-d346-4fae-9310-8e5da395bb9e-istiod-ca-cert\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.530784 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.530714    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-podinfo\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.530784 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.530781    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-workload-socket\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.530875 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.530813    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-data\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.530875 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.530832    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-envoy\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.631393 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.631364    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-workload-certs\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.631863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.631412    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-credential-socket\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.631863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.631439    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-token\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.631863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.631465    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3177cae4-d346-4fae-9310-8e5da395bb9e-istiod-ca-cert\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.631863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.631497    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-podinfo\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.631863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.631565    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-workload-socket\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.631863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.631603    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-data\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.631863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.631638    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-envoy\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.631863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.631680    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndcdq\" (UniqueName: \"kubernetes.io/projected/3177cae4-d346-4fae-9310-8e5da395bb9e-kube-api-access-ndcdq\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.631863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.631816    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-credential-socket\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.632312 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.632007    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-workload-certs\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.632312 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.632204    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-data\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.632312 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.632207    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-workload-socket\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.632312 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.632267    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3177cae4-d346-4fae-9310-8e5da395bb9e-istiod-ca-cert\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.634055 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.634038    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-envoy\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.634607 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.634584    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-podinfo\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.640519 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.640491    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-token\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.640615 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.640553    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndcdq\" (UniqueName: \"kubernetes.io/projected/3177cae4-d346-4fae-9310-8e5da395bb9e-kube-api-access-ndcdq\") pod \"e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") " pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:02.743420 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:02.743398    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:03.020760 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:03.020737    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"]
Sep 04 23:51:03.022597 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:51:03.022571    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3177cae4_d346_4fae_9310_8e5da395bb9e.slice/crio-d83c033d74dcdcf128f0c92aba612e6c1f0419287fe358a3faa672fa62102c99 WatchSource:0}: Error finding container d83c033d74dcdcf128f0c92aba612e6c1f0419287fe358a3faa672fa62102c99: Status 404 returned error can't find the container with id d83c033d74dcdcf128f0c92aba612e6c1f0419287fe358a3faa672fa62102c99
Sep 04 23:51:03.025005 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:03.024966    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:51:03.025065 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:03.025050    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:51:03.025109 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:03.025091    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:51:03.412719 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:03.412624    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9" event={"ID":"3177cae4-d346-4fae-9310-8e5da395bb9e","Type":"ContainerStarted","Data":"6bfa098d9f98a8d45ffed4336eaab50a8b97f46bc98d40d735af2e121b48e624"}
Sep 04 23:51:03.412719 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:03.412675    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9" event={"ID":"3177cae4-d346-4fae-9310-8e5da395bb9e","Type":"ContainerStarted","Data":"d83c033d74dcdcf128f0c92aba612e6c1f0419287fe358a3faa672fa62102c99"}
Sep 04 23:51:03.434871 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:03.434826    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9" podStartSLOduration=1.434812759 podStartE2EDuration="1.434812759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:51:03.433308572 +0000 UTC m=+845.851126891" watchObservedRunningTime="2026-09-04 23:51:03.434812759 +0000 UTC m=+845.852631077"
Sep 04 23:51:03.743774 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:03.743741    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:04.078962 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:04.078875    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08f394bb-2e0b-4101-954f-3d68b27aa216" path="/var/lib/kubelet/pods/08f394bb-2e0b-4101-954f-3d68b27aa216/volumes"
Sep 04 23:51:04.748524 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:04.748495    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:05.423428 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:05.423399    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:05.424519 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:05.424493    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:07.098384 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.098352    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"]
Sep 04 23:51:07.098724 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.098679    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08f394bb-2e0b-4101-954f-3d68b27aa216" containerName="istio-proxy"
Sep 04 23:51:07.098724 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.098689    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="08f394bb-2e0b-4101-954f-3d68b27aa216" containerName="istio-proxy"
Sep 04 23:51:07.098802 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.098755    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="08f394bb-2e0b-4101-954f-3d68b27aa216" containerName="istio-proxy"
Sep 04 23:51:07.101821 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.101805    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"
Sep 04 23:51:07.104303 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.104280    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-6fa8d6\""
Sep 04 23:51:07.104448 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.104286    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-6fa8d6-dockercfg-sqkfr\""
Sep 04 23:51:07.109500 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.109477    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"]
Sep 04 23:51:07.144634 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.144608    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"]
Sep 04 23:51:07.147744 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.147726    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"
Sep 04 23:51:07.161057 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.161026    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"]
Sep 04 23:51:07.170168 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.170137    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e1f71e3-ebe0-46d8-b304-efc76793a352-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q\" (UID: \"3e1f71e3-ebe0-46d8-b304-efc76793a352\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"
Sep 04 23:51:07.170288 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.170217    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j64sf\" (UniqueName: \"kubernetes.io/projected/3e1f71e3-ebe0-46d8-b304-efc76793a352-kube-api-access-j64sf\") pod \"payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q\" (UID: \"3e1f71e3-ebe0-46d8-b304-efc76793a352\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"
Sep 04 23:51:07.271024 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.270982    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e1f71e3-ebe0-46d8-b304-efc76793a352-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q\" (UID: \"3e1f71e3-ebe0-46d8-b304-efc76793a352\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"
Sep 04 23:51:07.271199 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.271082    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j64sf\" (UniqueName: \"kubernetes.io/projected/3e1f71e3-ebe0-46d8-b304-efc76793a352-kube-api-access-j64sf\") pod \"payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q\" (UID: \"3e1f71e3-ebe0-46d8-b304-efc76793a352\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"
Sep 04 23:51:07.271199 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.271110    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5twdh\" (UniqueName: \"kubernetes.io/projected/d2f7ec35-6894-4e26-b8ca-0dd8875b8e61-kube-api-access-5twdh\") pod \"payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq\" (UID: \"d2f7ec35-6894-4e26-b8ca-0dd8875b8e61\") " pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"
Sep 04 23:51:07.271199 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.271157    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f7ec35-6894-4e26-b8ca-0dd8875b8e61-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq\" (UID: \"d2f7ec35-6894-4e26-b8ca-0dd8875b8e61\") " pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"
Sep 04 23:51:07.271704 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.271684    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e1f71e3-ebe0-46d8-b304-efc76793a352-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q\" (UID: \"3e1f71e3-ebe0-46d8-b304-efc76793a352\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"
Sep 04 23:51:07.278618 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.278592    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j64sf\" (UniqueName: \"kubernetes.io/projected/3e1f71e3-ebe0-46d8-b304-efc76793a352-kube-api-access-j64sf\") pod \"payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q\" (UID: \"3e1f71e3-ebe0-46d8-b304-efc76793a352\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"
Sep 04 23:51:07.372424 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.372357    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5twdh\" (UniqueName: \"kubernetes.io/projected/d2f7ec35-6894-4e26-b8ca-0dd8875b8e61-kube-api-access-5twdh\") pod \"payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq\" (UID: \"d2f7ec35-6894-4e26-b8ca-0dd8875b8e61\") " pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"
Sep 04 23:51:07.372424 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.372398    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f7ec35-6894-4e26-b8ca-0dd8875b8e61-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq\" (UID: \"d2f7ec35-6894-4e26-b8ca-0dd8875b8e61\") " pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"
Sep 04 23:51:07.373036 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.373018    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f7ec35-6894-4e26-b8ca-0dd8875b8e61-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq\" (UID: \"d2f7ec35-6894-4e26-b8ca-0dd8875b8e61\") " pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"
Sep 04 23:51:07.380908 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.380884    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5twdh\" (UniqueName: \"kubernetes.io/projected/d2f7ec35-6894-4e26-b8ca-0dd8875b8e61-kube-api-access-5twdh\") pod \"payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq\" (UID: \"d2f7ec35-6894-4e26-b8ca-0dd8875b8e61\") " pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"
Sep 04 23:51:07.413767 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.413742    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"
Sep 04 23:51:07.462012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.461985    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"
Sep 04 23:51:07.686876 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.686855    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"]
Sep 04 23:51:07.688592 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:51:07.688565    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1f71e3_ebe0_46d8_b304_efc76793a352.slice/crio-4ec24f4f5b6bc7717f6f33eea834ba9a31dc68b0eea0b6e7561db7cea38f4a9a WatchSource:0}: Error finding container 4ec24f4f5b6bc7717f6f33eea834ba9a31dc68b0eea0b6e7561db7cea38f4a9a: Status 404 returned error can't find the container with id 4ec24f4f5b6bc7717f6f33eea834ba9a31dc68b0eea0b6e7561db7cea38f4a9a
Sep 04 23:51:07.745344 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:07.745319    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"]
Sep 04 23:51:07.747178 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:51:07.747155    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f7ec35_6894_4e26_b8ca_0dd8875b8e61.slice/crio-ef7fc42d6149a89bfff27f6d80dd0d93098d88164b5cbe90ec54a6c54e9e86fe WatchSource:0}: Error finding container ef7fc42d6149a89bfff27f6d80dd0d93098d88164b5cbe90ec54a6c54e9e86fe: Status 404 returned error can't find the container with id ef7fc42d6149a89bfff27f6d80dd0d93098d88164b5cbe90ec54a6c54e9e86fe
Sep 04 23:51:08.443117 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:08.443081    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q" event={"ID":"3e1f71e3-ebe0-46d8-b304-efc76793a352","Type":"ContainerStarted","Data":"1da71e280ff31e97a825d6c627864314f65063a107ddadee9569e83ea384d159"}
Sep 04 23:51:08.443569 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:08.443148    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q" event={"ID":"3e1f71e3-ebe0-46d8-b304-efc76793a352","Type":"ContainerStarted","Data":"4ec24f4f5b6bc7717f6f33eea834ba9a31dc68b0eea0b6e7561db7cea38f4a9a"}
Sep 04 23:51:08.443569 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:08.443204    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"
Sep 04 23:51:08.444967 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:08.444929    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq" event={"ID":"d2f7ec35-6894-4e26-b8ca-0dd8875b8e61","Type":"ContainerStarted","Data":"fb0c994848cf5d5ba4e1038e2ae0f4eb4da9f39b9ff457af4597826334f9f14e"}
Sep 04 23:51:08.445097 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:08.444982    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq" event={"ID":"d2f7ec35-6894-4e26-b8ca-0dd8875b8e61","Type":"ContainerStarted","Data":"ef7fc42d6149a89bfff27f6d80dd0d93098d88164b5cbe90ec54a6c54e9e86fe"}
Sep 04 23:51:08.445097 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:08.445019    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"
Sep 04 23:51:08.460641 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:08.460603    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q" podStartSLOduration=1.460590155 podStartE2EDuration="1.460590155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:51:08.458220509 +0000 UTC m=+850.876038835" watchObservedRunningTime="2026-09-04 23:51:08.460590155 +0000 UTC m=+850.878408474"
Sep 04 23:51:08.474240 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:08.474196    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq" podStartSLOduration=1.474178973 podStartE2EDuration="1.474178973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:51:08.472954113 +0000 UTC m=+850.890772435" watchObservedRunningTime="2026-09-04 23:51:08.474178973 +0000 UTC m=+850.891997292"
Sep 04 23:51:19.015242 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:19.015204    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:51:19.453002 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:19.452975    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"
Sep 04 23:51:19.453189 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:19.453054    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"
Sep 04 23:51:20.583911 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:20.583875    2557 patch_prober.go:28] interesting pod/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": read tcp 10.134.0.2:39842->10.134.0.50:15021: read: connection reset by peer" start-of-body=
Sep 04 23:51:20.584311 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:20.583958    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9" podUID="3177cae4-d346-4fae-9310-8e5da395bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": read tcp 10.134.0.2:39842->10.134.0.50:15021: read: connection reset by peer"
Sep 04 23:51:21.530606 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:21.530578    2557 generic.go:356] "Generic (PLEG): container finished" podID="3177cae4-d346-4fae-9310-8e5da395bb9e" containerID="6bfa098d9f98a8d45ffed4336eaab50a8b97f46bc98d40d735af2e121b48e624" exitCode=0
Sep 04 23:51:21.530779 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:21.530657    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9" event={"ID":"3177cae4-d346-4fae-9310-8e5da395bb9e","Type":"ContainerDied","Data":"6bfa098d9f98a8d45ffed4336eaab50a8b97f46bc98d40d735af2e121b48e624"}
Sep 04 23:51:21.531104 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:21.531088    2557 scope.go:117] "RemoveContainer" containerID="6bfa098d9f98a8d45ffed4336eaab50a8b97f46bc98d40d735af2e121b48e624"
Sep 04 23:51:21.532144 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:21.532112    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:51:21.532209 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:21.532198    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:51:21.532252 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:21.532232    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:51:22.538051 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:22.538015    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9" event={"ID":"3177cae4-d346-4fae-9310-8e5da395bb9e","Type":"ContainerStarted","Data":"303e5aa2ab55c5eb53f38c30693aeabf016b21b923b2516a3ec3513c00e4fcc6"}
Sep 04 23:51:22.743665 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:22.743630    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:22.743858 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:22.743809    2557 patch_prober.go:28] interesting pod/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 04 23:51:22.743923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:22.743867    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9" podUID="3177cae4-d346-4fae-9310-8e5da395bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 04 23:51:23.744218 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:23.744184    2557 patch_prober.go:28] interesting pod/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 04 23:51:23.744701 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:23.744270    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9" podUID="3177cae4-d346-4fae-9310-8e5da395bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 04 23:51:24.744571 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:24.744531    2557 patch_prober.go:28] interesting pod/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 04 23:51:24.744976 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:24.744594    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9" podUID="3177cae4-d346-4fae-9310-8e5da395bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 04 23:51:25.744657 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:25.744624    2557 patch_prober.go:28] interesting pod/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 04 23:51:25.745069 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:25.744677    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9" podUID="3177cae4-d346-4fae-9310-8e5da395bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 04 23:51:26.744305 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:26.744269    2557 patch_prober.go:28] interesting pod/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 04 23:51:26.744504 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:26.744324    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9" podUID="3177cae4-d346-4fae-9310-8e5da395bb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 04 23:51:27.747746 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:27.747711    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:27.748214 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:27.748195    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:27.748788 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:27.748769    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:51:32.624463 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:32.624405    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:51:48.612171 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:48.612138    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:51:56.817108 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:56.815214    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"]
Sep 04 23:51:56.818915 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:56.818888    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:56.822724 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:56.822701    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-6fa8d6\"/\"default-dockercfg-cjr85\""
Sep 04 23:51:56.824540 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:56.823367    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-6fa8d6\"/\"openshift-service-ca.crt\""
Sep 04 23:51:56.824540 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:56.823512    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-6fa8d6\"/\"kube-root-ca.crt\""
Sep 04 23:51:56.824540 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:56.823665    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-6fa8d6\"/\"explicit-model-6fa8d6-kserve-self-signed-certs\""
Sep 04 23:51:56.845031 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:56.844999    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"]
Sep 04 23:51:56.900567 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:56.900540    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-home\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:56.900680 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:56.900575    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsz5n\" (UniqueName: \"kubernetes.io/projected/691daf31-6700-465b-8c02-5ccad7e8d446-kube-api-access-lsz5n\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:56.900680 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:56.900664    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-model-cache\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:56.900790 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:56.900684    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-tmp-dir\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:56.900790 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:56.900705    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/691daf31-6700-465b-8c02-5ccad7e8d446-tls-certs\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:56.900790 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:56.900730    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-dshm\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:57.001627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.001596    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-home\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:57.001742 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.001630    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsz5n\" (UniqueName: \"kubernetes.io/projected/691daf31-6700-465b-8c02-5ccad7e8d446-kube-api-access-lsz5n\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:57.001742 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.001673    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-model-cache\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:57.001742 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.001691    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-tmp-dir\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:57.001742 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.001710    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/691daf31-6700-465b-8c02-5ccad7e8d446-tls-certs\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:57.001742 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.001727    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-dshm\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:57.002083 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.002063    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-home\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:57.002148 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.002101    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-tmp-dir\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:57.002212 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.002155    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-model-cache\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:57.004003 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.003984    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-dshm\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:57.004100 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.004084    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/691daf31-6700-465b-8c02-5ccad7e8d446-tls-certs\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:57.009041 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.009022    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsz5n\" (UniqueName: \"kubernetes.io/projected/691daf31-6700-465b-8c02-5ccad7e8d446-kube-api-access-lsz5n\") pod \"explicit-model-6fa8d6-kserve-84bb9f7d99-6578h\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") " pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:57.153565 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.153507    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:51:57.426471 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.426446    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"]
Sep 04 23:51:57.427891 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:51:57.427868    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod691daf31_6700_465b_8c02_5ccad7e8d446.slice/crio-b5537fdec8bf899ed3176b08231bb3cb1f2f85c521a6eca01deaa6b1bb970922 WatchSource:0}: Error finding container b5537fdec8bf899ed3176b08231bb3cb1f2f85c521a6eca01deaa6b1bb970922: Status 404 returned error can't find the container with id b5537fdec8bf899ed3176b08231bb3cb1f2f85c521a6eca01deaa6b1bb970922
Sep 04 23:51:57.739692 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.739659    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h" event={"ID":"691daf31-6700-465b-8c02-5ccad7e8d446","Type":"ContainerStarted","Data":"be1b8169f1921ae0608d1be0fbfe81bbd14f5cc28c5fee7a611b046c92a70d62"}
Sep 04 23:51:57.739850 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.739702    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h" event={"ID":"691daf31-6700-465b-8c02-5ccad7e8d446","Type":"ContainerStarted","Data":"b5537fdec8bf899ed3176b08231bb3cb1f2f85c521a6eca01deaa6b1bb970922"}
Sep 04 23:51:57.762590 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:57.762537    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h" podStartSLOduration=1.762523235 podStartE2EDuration="1.762523235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:51:57.759158059 +0000 UTC m=+900.176976378" watchObservedRunningTime="2026-09-04 23:51:57.762523235 +0000 UTC m=+900.180341556"
Sep 04 23:51:58.045967 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:58.045875    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 04 23:51:58.048856 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:51:58.048836    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 04 23:52:02.865554 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:02.865520    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"]
Sep 04 23:52:02.992776 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:02.992743    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz"]
Sep 04 23:52:02.996457 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:02.996435    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz"
Sep 04 23:52:02.999175 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:02.999142    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 23:52:02.999405 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:02.999381    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k4f64\""
Sep 04 23:52:03.003361 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:03.003338    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz"]
Sep 04 23:52:03.159719 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:03.159649    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5582a93-6cf0-439d-8943-b5978f93a00e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz\" (UID: \"b5582a93-6cf0-439d-8943-b5978f93a00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz"
Sep 04 23:52:03.260331 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:03.260305    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5582a93-6cf0-439d-8943-b5978f93a00e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz\" (UID: \"b5582a93-6cf0-439d-8943-b5978f93a00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz"
Sep 04 23:52:03.260836 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:03.260819    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5582a93-6cf0-439d-8943-b5978f93a00e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz\" (UID: \"b5582a93-6cf0-439d-8943-b5978f93a00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz"
Sep 04 23:52:03.309397 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:03.309376    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz"
Sep 04 23:52:03.519584 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:03.519554    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:52:03.577117 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:03.577096    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz"]
Sep 04 23:52:03.578388 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:52:03.578364    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5582a93_6cf0_439d_8943_b5978f93a00e.slice/crio-917bccf82951c48e6dec850e23413c8c683f2b445d627d1fc27a34a1c96a4a1a WatchSource:0}: Error finding container 917bccf82951c48e6dec850e23413c8c683f2b445d627d1fc27a34a1c96a4a1a: Status 404 returned error can't find the container with id 917bccf82951c48e6dec850e23413c8c683f2b445d627d1fc27a34a1c96a4a1a
Sep 04 23:52:03.773665 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:03.773611    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz" event={"ID":"b5582a93-6cf0-439d-8943-b5978f93a00e","Type":"ContainerStarted","Data":"917bccf82951c48e6dec850e23413c8c683f2b445d627d1fc27a34a1c96a4a1a"}
Sep 04 23:52:04.781413 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:04.781376    2557 generic.go:356] "Generic (PLEG): container finished" podID="b5582a93-6cf0-439d-8943-b5978f93a00e" containerID="cd0c61371a5a431ec319f864d1583f3cfd26ffeef202a4b1642a5eae4b4742ab" exitCode=0
Sep 04 23:52:04.781832 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:04.781456    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz" event={"ID":"b5582a93-6cf0-439d-8943-b5978f93a00e","Type":"ContainerDied","Data":"cd0c61371a5a431ec319f864d1583f3cfd26ffeef202a4b1642a5eae4b4742ab"}
Sep 04 23:52:05.912157 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:05.912134    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz"
Sep 04 23:52:06.090365 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:06.090301    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5582a93-6cf0-439d-8943-b5978f93a00e-maas-api-service-ca\") pod \"b5582a93-6cf0-439d-8943-b5978f93a00e\" (UID: \"b5582a93-6cf0-439d-8943-b5978f93a00e\") "
Sep 04 23:52:06.090623 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:06.090601    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5582a93-6cf0-439d-8943-b5978f93a00e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b5582a93-6cf0-439d-8943-b5978f93a00e" (UID: "b5582a93-6cf0-439d-8943-b5978f93a00e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:52:06.191759 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:06.191734    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5582a93-6cf0-439d-8943-b5978f93a00e-maas-api-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:06.793454 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:06.793427    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz"
Sep 04 23:52:06.793612 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:06.793470    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz" event={"ID":"b5582a93-6cf0-439d-8943-b5978f93a00e","Type":"ContainerDied","Data":"917bccf82951c48e6dec850e23413c8c683f2b445d627d1fc27a34a1c96a4a1a"}
Sep 04 23:52:06.793612 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:06.793514    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="917bccf82951c48e6dec850e23413c8c683f2b445d627d1fc27a34a1c96a4a1a"
Sep 04 23:52:07.154686 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:07.154602    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:52:13.451152 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.451122    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"]
Sep 04 23:52:13.451578 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.451344    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq" podUID="d2f7ec35-6894-4e26-b8ca-0dd8875b8e61" containerName="payload-processing" containerID="cri-o://fb0c994848cf5d5ba4e1038e2ae0f4eb4da9f39b9ff457af4597826334f9f14e" gracePeriod=30
Sep 04 23:52:13.457059 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.457033    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"]
Sep 04 23:52:13.457299 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.457252    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q" podUID="3e1f71e3-ebe0-46d8-b304-efc76793a352" containerName="payload-pre-processing" containerID="cri-o://1da71e280ff31e97a825d6c627864314f65063a107ddadee9569e83ea384d159" gracePeriod=30
Sep 04 23:52:13.597548 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.597512    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz"]
Sep 04 23:52:13.601593 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.601564    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-6fa8d6-gnkgz"]
Sep 04 23:52:13.736363 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.736338    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"
Sep 04 23:52:13.739554 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.739520    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"
Sep 04 23:52:13.835352 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.835319    2557 generic.go:356] "Generic (PLEG): container finished" podID="3e1f71e3-ebe0-46d8-b304-efc76793a352" containerID="1da71e280ff31e97a825d6c627864314f65063a107ddadee9569e83ea384d159" exitCode=0
Sep 04 23:52:13.835526 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.835363    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"
Sep 04 23:52:13.835526 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.835406    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q" event={"ID":"3e1f71e3-ebe0-46d8-b304-efc76793a352","Type":"ContainerDied","Data":"1da71e280ff31e97a825d6c627864314f65063a107ddadee9569e83ea384d159"}
Sep 04 23:52:13.835526 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.835468    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q" event={"ID":"3e1f71e3-ebe0-46d8-b304-efc76793a352","Type":"ContainerDied","Data":"4ec24f4f5b6bc7717f6f33eea834ba9a31dc68b0eea0b6e7561db7cea38f4a9a"}
Sep 04 23:52:13.835526 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.835497    2557 scope.go:117] "RemoveContainer" containerID="1da71e280ff31e97a825d6c627864314f65063a107ddadee9569e83ea384d159"
Sep 04 23:52:13.836642 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.836626    2557 generic.go:356] "Generic (PLEG): container finished" podID="d2f7ec35-6894-4e26-b8ca-0dd8875b8e61" containerID="fb0c994848cf5d5ba4e1038e2ae0f4eb4da9f39b9ff457af4597826334f9f14e" exitCode=0
Sep 04 23:52:13.836738 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.836661    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"
Sep 04 23:52:13.836738 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.836707    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq" event={"ID":"d2f7ec35-6894-4e26-b8ca-0dd8875b8e61","Type":"ContainerDied","Data":"fb0c994848cf5d5ba4e1038e2ae0f4eb4da9f39b9ff457af4597826334f9f14e"}
Sep 04 23:52:13.836825 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.836740    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq" event={"ID":"d2f7ec35-6894-4e26-b8ca-0dd8875b8e61","Type":"ContainerDied","Data":"ef7fc42d6149a89bfff27f6d80dd0d93098d88164b5cbe90ec54a6c54e9e86fe"}
Sep 04 23:52:13.846592 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.846558    2557 scope.go:117] "RemoveContainer" containerID="1da71e280ff31e97a825d6c627864314f65063a107ddadee9569e83ea384d159"
Sep 04 23:52:13.846979 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:52:13.846931    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da71e280ff31e97a825d6c627864314f65063a107ddadee9569e83ea384d159\": container with ID starting with 1da71e280ff31e97a825d6c627864314f65063a107ddadee9569e83ea384d159 not found: ID does not exist" containerID="1da71e280ff31e97a825d6c627864314f65063a107ddadee9569e83ea384d159"
Sep 04 23:52:13.847068 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.846991    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da71e280ff31e97a825d6c627864314f65063a107ddadee9569e83ea384d159"} err="failed to get container status \"1da71e280ff31e97a825d6c627864314f65063a107ddadee9569e83ea384d159\": rpc error: code = NotFound desc = could not find container \"1da71e280ff31e97a825d6c627864314f65063a107ddadee9569e83ea384d159\": container with ID starting with 1da71e280ff31e97a825d6c627864314f65063a107ddadee9569e83ea384d159 not found: ID does not exist"
Sep 04 23:52:13.847068 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.847013    2557 scope.go:117] "RemoveContainer" containerID="fb0c994848cf5d5ba4e1038e2ae0f4eb4da9f39b9ff457af4597826334f9f14e"
Sep 04 23:52:13.854784 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.854768    2557 scope.go:117] "RemoveContainer" containerID="fb0c994848cf5d5ba4e1038e2ae0f4eb4da9f39b9ff457af4597826334f9f14e"
Sep 04 23:52:13.855074 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.855054    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j64sf\" (UniqueName: \"kubernetes.io/projected/3e1f71e3-ebe0-46d8-b304-efc76793a352-kube-api-access-j64sf\") pod \"3e1f71e3-ebe0-46d8-b304-efc76793a352\" (UID: \"3e1f71e3-ebe0-46d8-b304-efc76793a352\") "
Sep 04 23:52:13.855183 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:52:13.855068    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0c994848cf5d5ba4e1038e2ae0f4eb4da9f39b9ff457af4597826334f9f14e\": container with ID starting with fb0c994848cf5d5ba4e1038e2ae0f4eb4da9f39b9ff457af4597826334f9f14e not found: ID does not exist" containerID="fb0c994848cf5d5ba4e1038e2ae0f4eb4da9f39b9ff457af4597826334f9f14e"
Sep 04 23:52:13.855183 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.855093    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0c994848cf5d5ba4e1038e2ae0f4eb4da9f39b9ff457af4597826334f9f14e"} err="failed to get container status \"fb0c994848cf5d5ba4e1038e2ae0f4eb4da9f39b9ff457af4597826334f9f14e\": rpc error: code = NotFound desc = could not find container \"fb0c994848cf5d5ba4e1038e2ae0f4eb4da9f39b9ff457af4597826334f9f14e\": container with ID starting with fb0c994848cf5d5ba4e1038e2ae0f4eb4da9f39b9ff457af4597826334f9f14e not found: ID does not exist"
Sep 04 23:52:13.855183 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.855141    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5twdh\" (UniqueName: \"kubernetes.io/projected/d2f7ec35-6894-4e26-b8ca-0dd8875b8e61-kube-api-access-5twdh\") pod \"d2f7ec35-6894-4e26-b8ca-0dd8875b8e61\" (UID: \"d2f7ec35-6894-4e26-b8ca-0dd8875b8e61\") "
Sep 04 23:52:13.855362 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.855218    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e1f71e3-ebe0-46d8-b304-efc76793a352-plugins-config-volume\") pod \"3e1f71e3-ebe0-46d8-b304-efc76793a352\" (UID: \"3e1f71e3-ebe0-46d8-b304-efc76793a352\") "
Sep 04 23:52:13.855362 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.855260    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f7ec35-6894-4e26-b8ca-0dd8875b8e61-plugins-config-volume\") pod \"d2f7ec35-6894-4e26-b8ca-0dd8875b8e61\" (UID: \"d2f7ec35-6894-4e26-b8ca-0dd8875b8e61\") "
Sep 04 23:52:13.855620 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.855596    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f7ec35-6894-4e26-b8ca-0dd8875b8e61-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2f7ec35-6894-4e26-b8ca-0dd8875b8e61" (UID: "d2f7ec35-6894-4e26-b8ca-0dd8875b8e61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:52:13.855779 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.855763    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1f71e3-ebe0-46d8-b304-efc76793a352-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e1f71e3-ebe0-46d8-b304-efc76793a352" (UID: "3e1f71e3-ebe0-46d8-b304-efc76793a352"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:52:13.856996 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.856976    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1f71e3-ebe0-46d8-b304-efc76793a352-kube-api-access-j64sf" (OuterVolumeSpecName: "kube-api-access-j64sf") pod "3e1f71e3-ebe0-46d8-b304-efc76793a352" (UID: "3e1f71e3-ebe0-46d8-b304-efc76793a352"). InnerVolumeSpecName "kube-api-access-j64sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:52:13.857165 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.857147    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f7ec35-6894-4e26-b8ca-0dd8875b8e61-kube-api-access-5twdh" (OuterVolumeSpecName: "kube-api-access-5twdh") pod "d2f7ec35-6894-4e26-b8ca-0dd8875b8e61" (UID: "d2f7ec35-6894-4e26-b8ca-0dd8875b8e61"). InnerVolumeSpecName "kube-api-access-5twdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:52:13.956098 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.956072    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5twdh\" (UniqueName: \"kubernetes.io/projected/d2f7ec35-6894-4e26-b8ca-0dd8875b8e61-kube-api-access-5twdh\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:13.956098 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.956097    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e1f71e3-ebe0-46d8-b304-efc76793a352-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:13.956219 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.956106    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f7ec35-6894-4e26-b8ca-0dd8875b8e61-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:13.956219 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:13.956114    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j64sf\" (UniqueName: \"kubernetes.io/projected/3e1f71e3-ebe0-46d8-b304-efc76793a352-kube-api-access-j64sf\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:14.079528 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:14.079470    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5582a93-6cf0-439d-8943-b5978f93a00e" path="/var/lib/kubelet/pods/b5582a93-6cf0-439d-8943-b5978f93a00e/volumes"
Sep 04 23:52:14.152453 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:14.152422    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"]
Sep 04 23:52:14.156345 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:14.156324    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6fa8d6-56b5c776fd-mqx6q"]
Sep 04 23:52:14.167375 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:14.167351    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"]
Sep 04 23:52:14.169320 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:14.169302    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6fa8d6-85b657f95c-zv8tq"]
Sep 04 23:52:16.079823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:16.079786    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e1f71e3-ebe0-46d8-b304-efc76793a352" path="/var/lib/kubelet/pods/3e1f71e3-ebe0-46d8-b304-efc76793a352/volumes"
Sep 04 23:52:16.080248 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:16.080226    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f7ec35-6894-4e26-b8ca-0dd8875b8e61" path="/var/lib/kubelet/pods/d2f7ec35-6894-4e26-b8ca-0dd8875b8e61/volumes"
Sep 04 23:52:17.718311 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:17.718270    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:52:17.885434 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:17.885380    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h" podUID="691daf31-6700-465b-8c02-5ccad7e8d446" containerName="main" containerID="cri-o://be1b8169f1921ae0608d1be0fbfe81bbd14f5cc28c5fee7a611b046c92a70d62" gracePeriod=45
Sep 04 23:52:18.132839 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.132816    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:52:18.191475 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.191446    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-tmp-dir\") pod \"691daf31-6700-465b-8c02-5ccad7e8d446\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") "
Sep 04 23:52:18.191610 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.191535    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/691daf31-6700-465b-8c02-5ccad7e8d446-tls-certs\") pod \"691daf31-6700-465b-8c02-5ccad7e8d446\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") "
Sep 04 23:52:18.191610 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.191570    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-dshm\") pod \"691daf31-6700-465b-8c02-5ccad7e8d446\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") "
Sep 04 23:52:18.191610 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.191600    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-model-cache\") pod \"691daf31-6700-465b-8c02-5ccad7e8d446\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") "
Sep 04 23:52:18.191768 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.191654    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-home\") pod \"691daf31-6700-465b-8c02-5ccad7e8d446\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") "
Sep 04 23:52:18.191768 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.191699    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsz5n\" (UniqueName: \"kubernetes.io/projected/691daf31-6700-465b-8c02-5ccad7e8d446-kube-api-access-lsz5n\") pod \"691daf31-6700-465b-8c02-5ccad7e8d446\" (UID: \"691daf31-6700-465b-8c02-5ccad7e8d446\") "
Sep 04 23:52:18.191768 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.191712    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "691daf31-6700-465b-8c02-5ccad7e8d446" (UID: "691daf31-6700-465b-8c02-5ccad7e8d446"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:18.191992 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.191957    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-model-cache" (OuterVolumeSpecName: "model-cache") pod "691daf31-6700-465b-8c02-5ccad7e8d446" (UID: "691daf31-6700-465b-8c02-5ccad7e8d446"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:18.191992 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.191981    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-home" (OuterVolumeSpecName: "home") pod "691daf31-6700-465b-8c02-5ccad7e8d446" (UID: "691daf31-6700-465b-8c02-5ccad7e8d446"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:18.192200 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.192113    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:18.192200 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.192134    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:18.192200 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.192147    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:18.194598 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.194507    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691daf31-6700-465b-8c02-5ccad7e8d446-kube-api-access-lsz5n" (OuterVolumeSpecName: "kube-api-access-lsz5n") pod "691daf31-6700-465b-8c02-5ccad7e8d446" (UID: "691daf31-6700-465b-8c02-5ccad7e8d446"). InnerVolumeSpecName "kube-api-access-lsz5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:52:18.194598 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.194554    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-dshm" (OuterVolumeSpecName: "dshm") pod "691daf31-6700-465b-8c02-5ccad7e8d446" (UID: "691daf31-6700-465b-8c02-5ccad7e8d446"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:18.194598 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.194562    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/691daf31-6700-465b-8c02-5ccad7e8d446-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "691daf31-6700-465b-8c02-5ccad7e8d446" (UID: "691daf31-6700-465b-8c02-5ccad7e8d446"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:52:18.293168 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.293085    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsz5n\" (UniqueName: \"kubernetes.io/projected/691daf31-6700-465b-8c02-5ccad7e8d446-kube-api-access-lsz5n\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:18.293168 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.293113    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/691daf31-6700-465b-8c02-5ccad7e8d446-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:18.293168 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.293123    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/691daf31-6700-465b-8c02-5ccad7e8d446-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:18.874643 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.874609    2557 generic.go:356] "Generic (PLEG): container finished" podID="691daf31-6700-465b-8c02-5ccad7e8d446" containerID="be1b8169f1921ae0608d1be0fbfe81bbd14f5cc28c5fee7a611b046c92a70d62" exitCode=0
Sep 04 23:52:18.875056 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.874652    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"
Sep 04 23:52:18.875056 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.874684    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h" event={"ID":"691daf31-6700-465b-8c02-5ccad7e8d446","Type":"ContainerDied","Data":"be1b8169f1921ae0608d1be0fbfe81bbd14f5cc28c5fee7a611b046c92a70d62"}
Sep 04 23:52:18.875056 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.874724    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h" event={"ID":"691daf31-6700-465b-8c02-5ccad7e8d446","Type":"ContainerDied","Data":"b5537fdec8bf899ed3176b08231bb3cb1f2f85c521a6eca01deaa6b1bb970922"}
Sep 04 23:52:18.875056 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.874741    2557 scope.go:117] "RemoveContainer" containerID="be1b8169f1921ae0608d1be0fbfe81bbd14f5cc28c5fee7a611b046c92a70d62"
Sep 04 23:52:18.885392 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.885373    2557 scope.go:117] "RemoveContainer" containerID="be1b8169f1921ae0608d1be0fbfe81bbd14f5cc28c5fee7a611b046c92a70d62"
Sep 04 23:52:18.885679 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:52:18.885660    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1b8169f1921ae0608d1be0fbfe81bbd14f5cc28c5fee7a611b046c92a70d62\": container with ID starting with be1b8169f1921ae0608d1be0fbfe81bbd14f5cc28c5fee7a611b046c92a70d62 not found: ID does not exist" containerID="be1b8169f1921ae0608d1be0fbfe81bbd14f5cc28c5fee7a611b046c92a70d62"
Sep 04 23:52:18.885731 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.885689    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1b8169f1921ae0608d1be0fbfe81bbd14f5cc28c5fee7a611b046c92a70d62"} err="failed to get container status \"be1b8169f1921ae0608d1be0fbfe81bbd14f5cc28c5fee7a611b046c92a70d62\": rpc error: code = NotFound desc = could not find container \"be1b8169f1921ae0608d1be0fbfe81bbd14f5cc28c5fee7a611b046c92a70d62\": container with ID starting with be1b8169f1921ae0608d1be0fbfe81bbd14f5cc28c5fee7a611b046c92a70d62 not found: ID does not exist"
Sep 04 23:52:18.897097 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.897075    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"]
Sep 04 23:52:18.902397 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:18.902376    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-6fa8d6/explicit-model-6fa8d6-kserve-84bb9f7d99-6578h"]
Sep 04 23:52:20.026493 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:20.026464    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"]
Sep 04 23:52:20.026966 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:20.026715    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9" podUID="3177cae4-d346-4fae-9310-8e5da395bb9e" containerName="istio-proxy" containerID="cri-o://303e5aa2ab55c5eb53f38c30693aeabf016b21b923b2516a3ec3513c00e4fcc6" gracePeriod=30
Sep 04 23:52:20.084846 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:20.084815    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="691daf31-6700-465b-8c02-5ccad7e8d446" path="/var/lib/kubelet/pods/691daf31-6700-465b-8c02-5ccad7e8d446/volumes"
Sep 04 23:52:25.294907 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.294853    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:52:25.349807 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.349763    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-workload-socket\") pod \"3177cae4-d346-4fae-9310-8e5da395bb9e\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") "
Sep 04 23:52:25.349807 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.349808    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-data\") pod \"3177cae4-d346-4fae-9310-8e5da395bb9e\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") "
Sep 04 23:52:25.350024 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.349838    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-podinfo\") pod \"3177cae4-d346-4fae-9310-8e5da395bb9e\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") "
Sep 04 23:52:25.350024 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.349913    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-envoy\") pod \"3177cae4-d346-4fae-9310-8e5da395bb9e\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") "
Sep 04 23:52:25.350024 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.349982    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-workload-certs\") pod \"3177cae4-d346-4fae-9310-8e5da395bb9e\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") "
Sep 04 23:52:25.350024 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.350013    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-credential-socket\") pod \"3177cae4-d346-4fae-9310-8e5da395bb9e\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") "
Sep 04 23:52:25.350231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.350035    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-token\") pod \"3177cae4-d346-4fae-9310-8e5da395bb9e\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") "
Sep 04 23:52:25.350231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.350068    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndcdq\" (UniqueName: \"kubernetes.io/projected/3177cae4-d346-4fae-9310-8e5da395bb9e-kube-api-access-ndcdq\") pod \"3177cae4-d346-4fae-9310-8e5da395bb9e\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") "
Sep 04 23:52:25.350231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.350065    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3177cae4-d346-4fae-9310-8e5da395bb9e" (UID: "3177cae4-d346-4fae-9310-8e5da395bb9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:25.350231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.350087    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-data" (OuterVolumeSpecName: "istio-data") pod "3177cae4-d346-4fae-9310-8e5da395bb9e" (UID: "3177cae4-d346-4fae-9310-8e5da395bb9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:25.350231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.350137    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3177cae4-d346-4fae-9310-8e5da395bb9e-istiod-ca-cert\") pod \"3177cae4-d346-4fae-9310-8e5da395bb9e\" (UID: \"3177cae4-d346-4fae-9310-8e5da395bb9e\") "
Sep 04 23:52:25.350441 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.350281    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3177cae4-d346-4fae-9310-8e5da395bb9e" (UID: "3177cae4-d346-4fae-9310-8e5da395bb9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:25.350441 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.350294    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3177cae4-d346-4fae-9310-8e5da395bb9e" (UID: "3177cae4-d346-4fae-9310-8e5da395bb9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:25.350566 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.350503    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:25.350566 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.350524    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:25.350566 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.350538    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:25.350566 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.350551    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:25.350699 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.350562    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3177cae4-d346-4fae-9310-8e5da395bb9e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3177cae4-d346-4fae-9310-8e5da395bb9e" (UID: "3177cae4-d346-4fae-9310-8e5da395bb9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:52:25.352235 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.352211    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-token" (OuterVolumeSpecName: "istio-token") pod "3177cae4-d346-4fae-9310-8e5da395bb9e" (UID: "3177cae4-d346-4fae-9310-8e5da395bb9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:52:25.352350 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.352266    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3177cae4-d346-4fae-9310-8e5da395bb9e-kube-api-access-ndcdq" (OuterVolumeSpecName: "kube-api-access-ndcdq") pod "3177cae4-d346-4fae-9310-8e5da395bb9e" (UID: "3177cae4-d346-4fae-9310-8e5da395bb9e"). InnerVolumeSpecName "kube-api-access-ndcdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:52:25.352463 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.352434    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3177cae4-d346-4fae-9310-8e5da395bb9e" (UID: "3177cae4-d346-4fae-9310-8e5da395bb9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:25.352525 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.352435    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3177cae4-d346-4fae-9310-8e5da395bb9e" (UID: "3177cae4-d346-4fae-9310-8e5da395bb9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:52:25.451118 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.451087    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:25.451118 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.451120    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndcdq\" (UniqueName: \"kubernetes.io/projected/3177cae4-d346-4fae-9310-8e5da395bb9e-kube-api-access-ndcdq\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:25.451266 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.451133    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3177cae4-d346-4fae-9310-8e5da395bb9e-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:25.451266 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.451145    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:25.451266 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.451156    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3177cae4-d346-4fae-9310-8e5da395bb9e-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:25.918353 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.918318    2557 generic.go:356] "Generic (PLEG): container finished" podID="3177cae4-d346-4fae-9310-8e5da395bb9e" containerID="303e5aa2ab55c5eb53f38c30693aeabf016b21b923b2516a3ec3513c00e4fcc6" exitCode=0
Sep 04 23:52:25.918505 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.918365    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"
Sep 04 23:52:25.918505 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.918402    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9" event={"ID":"3177cae4-d346-4fae-9310-8e5da395bb9e","Type":"ContainerDied","Data":"303e5aa2ab55c5eb53f38c30693aeabf016b21b923b2516a3ec3513c00e4fcc6"}
Sep 04 23:52:25.918505 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.918459    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9" event={"ID":"3177cae4-d346-4fae-9310-8e5da395bb9e","Type":"ContainerDied","Data":"d83c033d74dcdcf128f0c92aba612e6c1f0419287fe358a3faa672fa62102c99"}
Sep 04 23:52:25.918505 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.918483    2557 scope.go:117] "RemoveContainer" containerID="303e5aa2ab55c5eb53f38c30693aeabf016b21b923b2516a3ec3513c00e4fcc6"
Sep 04 23:52:25.927523 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.927506    2557 scope.go:117] "RemoveContainer" containerID="6bfa098d9f98a8d45ffed4336eaab50a8b97f46bc98d40d735af2e121b48e624"
Sep 04 23:52:25.935631 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.935613    2557 scope.go:117] "RemoveContainer" containerID="303e5aa2ab55c5eb53f38c30693aeabf016b21b923b2516a3ec3513c00e4fcc6"
Sep 04 23:52:25.935883 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:52:25.935861    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303e5aa2ab55c5eb53f38c30693aeabf016b21b923b2516a3ec3513c00e4fcc6\": container with ID starting with 303e5aa2ab55c5eb53f38c30693aeabf016b21b923b2516a3ec3513c00e4fcc6 not found: ID does not exist" containerID="303e5aa2ab55c5eb53f38c30693aeabf016b21b923b2516a3ec3513c00e4fcc6"
Sep 04 23:52:25.935987 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.935888    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303e5aa2ab55c5eb53f38c30693aeabf016b21b923b2516a3ec3513c00e4fcc6"} err="failed to get container status \"303e5aa2ab55c5eb53f38c30693aeabf016b21b923b2516a3ec3513c00e4fcc6\": rpc error: code = NotFound desc = could not find container \"303e5aa2ab55c5eb53f38c30693aeabf016b21b923b2516a3ec3513c00e4fcc6\": container with ID starting with 303e5aa2ab55c5eb53f38c30693aeabf016b21b923b2516a3ec3513c00e4fcc6 not found: ID does not exist"
Sep 04 23:52:25.935987 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.935906    2557 scope.go:117] "RemoveContainer" containerID="6bfa098d9f98a8d45ffed4336eaab50a8b97f46bc98d40d735af2e121b48e624"
Sep 04 23:52:25.936182 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:52:25.936165    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfa098d9f98a8d45ffed4336eaab50a8b97f46bc98d40d735af2e121b48e624\": container with ID starting with 6bfa098d9f98a8d45ffed4336eaab50a8b97f46bc98d40d735af2e121b48e624 not found: ID does not exist" containerID="6bfa098d9f98a8d45ffed4336eaab50a8b97f46bc98d40d735af2e121b48e624"
Sep 04 23:52:25.936227 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.936188    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfa098d9f98a8d45ffed4336eaab50a8b97f46bc98d40d735af2e121b48e624"} err="failed to get container status \"6bfa098d9f98a8d45ffed4336eaab50a8b97f46bc98d40d735af2e121b48e624\": rpc error: code = NotFound desc = could not find container \"6bfa098d9f98a8d45ffed4336eaab50a8b97f46bc98d40d735af2e121b48e624\": container with ID starting with 6bfa098d9f98a8d45ffed4336eaab50a8b97f46bc98d40d735af2e121b48e624 not found: ID does not exist"
Sep 04 23:52:25.942408 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.942374    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"]
Sep 04 23:52:25.947593 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:25.947571    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-6fa8d6-openshift-default-fbb756f44-5hwn9"]
Sep 04 23:52:26.078582 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:26.078552    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3177cae4-d346-4fae-9310-8e5da395bb9e" path="/var/lib/kubelet/pods/3177cae4-d346-4fae-9310-8e5da395bb9e/volumes"
Sep 04 23:52:30.700038 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700004    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"]
Sep 04 23:52:30.700555 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700535    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3177cae4-d346-4fae-9310-8e5da395bb9e" containerName="istio-proxy"
Sep 04 23:52:30.700604 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700560    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3177cae4-d346-4fae-9310-8e5da395bb9e" containerName="istio-proxy"
Sep 04 23:52:30.700604 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700572    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1f71e3-ebe0-46d8-b304-efc76793a352" containerName="payload-pre-processing"
Sep 04 23:52:30.700604 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700581    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1f71e3-ebe0-46d8-b304-efc76793a352" containerName="payload-pre-processing"
Sep 04 23:52:30.700698 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700608    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691daf31-6700-465b-8c02-5ccad7e8d446" containerName="main"
Sep 04 23:52:30.700698 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700617    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="691daf31-6700-465b-8c02-5ccad7e8d446" containerName="main"
Sep 04 23:52:30.700698 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700642    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3177cae4-d346-4fae-9310-8e5da395bb9e" containerName="istio-proxy"
Sep 04 23:52:30.700698 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700650    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3177cae4-d346-4fae-9310-8e5da395bb9e" containerName="istio-proxy"
Sep 04 23:52:30.700698 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700660    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5582a93-6cf0-439d-8943-b5978f93a00e" containerName="revoke-keys"
Sep 04 23:52:30.700698 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700669    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5582a93-6cf0-439d-8943-b5978f93a00e" containerName="revoke-keys"
Sep 04 23:52:30.700698 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700683    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f7ec35-6894-4e26-b8ca-0dd8875b8e61" containerName="payload-processing"
Sep 04 23:52:30.700698 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700690    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f7ec35-6894-4e26-b8ca-0dd8875b8e61" containerName="payload-processing"
Sep 04 23:52:30.700963 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700777    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="691daf31-6700-465b-8c02-5ccad7e8d446" containerName="main"
Sep 04 23:52:30.700963 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700790    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f7ec35-6894-4e26-b8ca-0dd8875b8e61" containerName="payload-processing"
Sep 04 23:52:30.700963 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700800    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3177cae4-d346-4fae-9310-8e5da395bb9e" containerName="istio-proxy"
Sep 04 23:52:30.700963 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700812    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3177cae4-d346-4fae-9310-8e5da395bb9e" containerName="istio-proxy"
Sep 04 23:52:30.700963 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700821    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5582a93-6cf0-439d-8943-b5978f93a00e" containerName="revoke-keys"
Sep 04 23:52:30.700963 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.700832    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e1f71e3-ebe0-46d8-b304-efc76793a352" containerName="payload-pre-processing"
Sep 04 23:52:30.704427 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.704404    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.707101 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.707079    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f4c99d38-openshift-default-dockercfg-lvj6w\""
Sep 04 23:52:30.715069 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.715047    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"]
Sep 04 23:52:30.790346 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.790308    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhlzf\" (UniqueName: \"kubernetes.io/projected/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-kube-api-access-qhlzf\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.790538 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.790375    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-podinfo\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.790538 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.790404    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-workload-socket\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.790538 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.790483    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-credential-socket\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.790718 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.790536    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-token\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.790718 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.790582    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-workload-certs\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.790718 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.790605    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istiod-ca-cert\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.790718 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.790650    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-envoy\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.790718 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.790698    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-data\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.891244 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.891214    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-credential-socket\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.891436 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.891264    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-token\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.891436 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.891307    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-workload-certs\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.891436 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.891330    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istiod-ca-cert\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.891436 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.891370    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-envoy\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.891666 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.891500    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-data\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.891666 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.891559    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhlzf\" (UniqueName: \"kubernetes.io/projected/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-kube-api-access-qhlzf\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.891666 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.891604    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-podinfo\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.891666 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.891631    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-workload-socket\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.891666 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.891655    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-credential-socket\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.891900 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.891758    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-workload-certs\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.892064 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.892044    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-workload-socket\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.892164 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.892144    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istiod-ca-cert\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.892204 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.892159    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-data\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.893596 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.893576    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-envoy\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.893922 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.893905    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-podinfo\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.901293 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.901242    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhlzf\" (UniqueName: \"kubernetes.io/projected/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-kube-api-access-qhlzf\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:30.901293 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:30.901254    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-token\") pod \"e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") " pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:31.015929 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:31.015872    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:31.293999 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:31.293971    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"]
Sep 04 23:52:31.295372 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:52:31.295348    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3220ae_ffa8_4ba2_95fd_7127685cf6fb.slice/crio-4f09149a254cbddd91226a20b7b359932c660e901c3c5cbc64b34ac45fc4307f WatchSource:0}: Error finding container 4f09149a254cbddd91226a20b7b359932c660e901c3c5cbc64b34ac45fc4307f: Status 404 returned error can't find the container with id 4f09149a254cbddd91226a20b7b359932c660e901c3c5cbc64b34ac45fc4307f
Sep 04 23:52:31.297491 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:31.297458    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:52:31.297592 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:31.297534    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:52:31.297592 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:31.297577    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:52:31.953706 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:31.953675    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl" event={"ID":"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb","Type":"ContainerStarted","Data":"0e00a47fa87ee2b52c0b97bf47ffc4e8ebbd01abf974f03118a7368d2dcb7266"}
Sep 04 23:52:31.954077 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:31.953728    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl" event={"ID":"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb","Type":"ContainerStarted","Data":"4f09149a254cbddd91226a20b7b359932c660e901c3c5cbc64b34ac45fc4307f"}
Sep 04 23:52:31.977283 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:31.977233    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl" podStartSLOduration=1.977219286 podStartE2EDuration="1.977219286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:52:31.976307021 +0000 UTC m=+934.394125340" watchObservedRunningTime="2026-09-04 23:52:31.977219286 +0000 UTC m=+934.395037598"
Sep 04 23:52:32.016285 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.016260    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:32.021389 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.021371    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:32.022854 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.022835    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"]
Sep 04 23:52:32.025422 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.025408    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.027789 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.027769    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ffeb50\"/\"default-dockercfg-pr7m8\""
Sep 04 23:52:32.027900 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.027881    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ffeb50\"/\"openshift-service-ca.crt\""
Sep 04 23:52:32.027900 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.027895    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ffeb50\"/\"noten-model-ffeb50-kserve-self-signed-certs\""
Sep 04 23:52:32.028039 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.027896    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ffeb50\"/\"kube-root-ca.crt\""
Sep 04 23:52:32.036866 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.036840    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"]
Sep 04 23:52:32.102196 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.102167    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-model-cache\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.102323 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.102229    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303b0311-0cb1-4ab8-8af2-9712e81624b9-tls-certs\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.102323 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.102291    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-dshm\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.102419 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.102342    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-tmp-dir\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.102419 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.102360    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw6q6\" (UniqueName: \"kubernetes.io/projected/303b0311-0cb1-4ab8-8af2-9712e81624b9-kube-api-access-kw6q6\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.102419 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.102407    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-home\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.203397 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.203362    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-tmp-dir\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.203565 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.203401    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw6q6\" (UniqueName: \"kubernetes.io/projected/303b0311-0cb1-4ab8-8af2-9712e81624b9-kube-api-access-kw6q6\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.203565 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.203470    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-home\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.203565 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.203502    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-model-cache\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.203744 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.203583    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303b0311-0cb1-4ab8-8af2-9712e81624b9-tls-certs\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.203744 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.203613    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-dshm\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.203874 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.203828    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-tmp-dir\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.203874 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.203862    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-home\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.204028 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.203983    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-model-cache\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.205802 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.205775    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-dshm\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.205968 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.205932    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303b0311-0cb1-4ab8-8af2-9712e81624b9-tls-certs\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.210798 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.210775    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw6q6\" (UniqueName: \"kubernetes.io/projected/303b0311-0cb1-4ab8-8af2-9712e81624b9-kube-api-access-kw6q6\") pod \"noten-model-ffeb50-kserve-5c4cf88f59-7hzhl\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") " pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.337946 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.337922    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:32.606901 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.606877    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"]
Sep 04 23:52:32.608841 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:52:32.608817    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod303b0311_0cb1_4ab8_8af2_9712e81624b9.slice/crio-baaa80e73f59e5fb345557f4f2406e8fd642b9e54bca9ae34ac29ad870c15eff WatchSource:0}: Error finding container baaa80e73f59e5fb345557f4f2406e8fd642b9e54bca9ae34ac29ad870c15eff: Status 404 returned error can't find the container with id baaa80e73f59e5fb345557f4f2406e8fd642b9e54bca9ae34ac29ad870c15eff
Sep 04 23:52:32.961804 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.961771    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl" event={"ID":"303b0311-0cb1-4ab8-8af2-9712e81624b9","Type":"ContainerStarted","Data":"b4e56eb30bb3286b30a1daad941bcec2a5baabca8b61f1a42544ba96b0ac5748"}
Sep 04 23:52:32.962234 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.961812    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl" event={"ID":"303b0311-0cb1-4ab8-8af2-9712e81624b9","Type":"ContainerStarted","Data":"baaa80e73f59e5fb345557f4f2406e8fd642b9e54bca9ae34ac29ad870c15eff"}
Sep 04 23:52:32.962317 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.962238    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:32.963172 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.963154    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:32.979970 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:32.979907    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl" podStartSLOduration=0.979895777 podStartE2EDuration="979.895777ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:52:32.978740197 +0000 UTC m=+935.396558579" watchObservedRunningTime="2026-09-04 23:52:32.979895777 +0000 UTC m=+935.397714095"
Sep 04 23:52:35.597502 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.597470    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"]
Sep 04 23:52:35.774210 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.774178    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"]
Sep 04 23:52:35.776608 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.776591    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"
Sep 04 23:52:35.779221 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.779202    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f4c99d38\""
Sep 04 23:52:35.779324 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.779233    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f4c99d38-dockercfg-4sstt\""
Sep 04 23:52:35.787566 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.787546    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"]
Sep 04 23:52:35.796061 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.796028    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"]
Sep 04 23:52:35.798749 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.798732    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"
Sep 04 23:52:35.811908 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.811887    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"]
Sep 04 23:52:35.835514 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.835489    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc2j4\" (UniqueName: \"kubernetes.io/projected/44b6b6f1-ab7a-4263-8b88-37242d37fac2-kube-api-access-rc2j4\") pod \"payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m\" (UID: \"44b6b6f1-ab7a-4263-8b88-37242d37fac2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"
Sep 04 23:52:35.835614 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.835540    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b6b6f1-ab7a-4263-8b88-37242d37fac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m\" (UID: \"44b6b6f1-ab7a-4263-8b88-37242d37fac2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"
Sep 04 23:52:35.835614 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.835579    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2lxf\" (UniqueName: \"kubernetes.io/projected/f7189bf6-8406-4fb5-b06c-d8e60f7501d2-kube-api-access-r2lxf\") pod \"payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q\" (UID: \"f7189bf6-8406-4fb5-b06c-d8e60f7501d2\") " pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"
Sep 04 23:52:35.835688 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.835632    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7189bf6-8406-4fb5-b06c-d8e60f7501d2-plugins-config-volume\") pod \"payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q\" (UID: \"f7189bf6-8406-4fb5-b06c-d8e60f7501d2\") " pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"
Sep 04 23:52:35.936710 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.936623    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7189bf6-8406-4fb5-b06c-d8e60f7501d2-plugins-config-volume\") pod \"payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q\" (UID: \"f7189bf6-8406-4fb5-b06c-d8e60f7501d2\") " pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"
Sep 04 23:52:35.936710 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.936668    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc2j4\" (UniqueName: \"kubernetes.io/projected/44b6b6f1-ab7a-4263-8b88-37242d37fac2-kube-api-access-rc2j4\") pod \"payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m\" (UID: \"44b6b6f1-ab7a-4263-8b88-37242d37fac2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"
Sep 04 23:52:35.936710 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.936712    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b6b6f1-ab7a-4263-8b88-37242d37fac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m\" (UID: \"44b6b6f1-ab7a-4263-8b88-37242d37fac2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"
Sep 04 23:52:35.937013 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.936756    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2lxf\" (UniqueName: \"kubernetes.io/projected/f7189bf6-8406-4fb5-b06c-d8e60f7501d2-kube-api-access-r2lxf\") pod \"payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q\" (UID: \"f7189bf6-8406-4fb5-b06c-d8e60f7501d2\") " pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"
Sep 04 23:52:35.937467 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.937440    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7189bf6-8406-4fb5-b06c-d8e60f7501d2-plugins-config-volume\") pod \"payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q\" (UID: \"f7189bf6-8406-4fb5-b06c-d8e60f7501d2\") " pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"
Sep 04 23:52:35.937577 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.937489    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b6b6f1-ab7a-4263-8b88-37242d37fac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m\" (UID: \"44b6b6f1-ab7a-4263-8b88-37242d37fac2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"
Sep 04 23:52:35.944473 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.944450    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc2j4\" (UniqueName: \"kubernetes.io/projected/44b6b6f1-ab7a-4263-8b88-37242d37fac2-kube-api-access-rc2j4\") pod \"payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m\" (UID: \"44b6b6f1-ab7a-4263-8b88-37242d37fac2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"
Sep 04 23:52:35.944669 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:35.944648    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2lxf\" (UniqueName: \"kubernetes.io/projected/f7189bf6-8406-4fb5-b06c-d8e60f7501d2-kube-api-access-r2lxf\") pod \"payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q\" (UID: \"f7189bf6-8406-4fb5-b06c-d8e60f7501d2\") " pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"
Sep 04 23:52:36.028198 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:36.028161    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:52:36.087227 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:36.087202    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"
Sep 04 23:52:36.108608 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:36.108570    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"
Sep 04 23:52:36.360898 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:36.360874    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"]
Sep 04 23:52:36.362433 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:52:36.362405    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b6b6f1_ab7a_4263_8b88_37242d37fac2.slice/crio-a46be7c8a1bfb5981086ddec17f0acdfc83a26c71aa20ea2ca092ad8f6bd2782 WatchSource:0}: Error finding container a46be7c8a1bfb5981086ddec17f0acdfc83a26c71aa20ea2ca092ad8f6bd2782: Status 404 returned error can't find the container with id a46be7c8a1bfb5981086ddec17f0acdfc83a26c71aa20ea2ca092ad8f6bd2782
Sep 04 23:52:36.396910 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:36.396887    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"]
Sep 04 23:52:36.398151 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:52:36.397987    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7189bf6_8406_4fb5_b06c_d8e60f7501d2.slice/crio-cfd3d65860098cb9ec3e2d7b13509a4957b8f0d0811dd3e544195c40dbbee91b WatchSource:0}: Error finding container cfd3d65860098cb9ec3e2d7b13509a4957b8f0d0811dd3e544195c40dbbee91b: Status 404 returned error can't find the container with id cfd3d65860098cb9ec3e2d7b13509a4957b8f0d0811dd3e544195c40dbbee91b
Sep 04 23:52:36.988042 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:36.988009    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q" event={"ID":"f7189bf6-8406-4fb5-b06c-d8e60f7501d2","Type":"ContainerStarted","Data":"3864a618ecd409961fa6defd4e6b431c132c2f3f220d21c6a585fab269fe53f3"}
Sep 04 23:52:36.988512 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:36.988076    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"
Sep 04 23:52:36.988512 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:36.988101    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q" event={"ID":"f7189bf6-8406-4fb5-b06c-d8e60f7501d2","Type":"ContainerStarted","Data":"cfd3d65860098cb9ec3e2d7b13509a4957b8f0d0811dd3e544195c40dbbee91b"}
Sep 04 23:52:36.989709 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:36.989684    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m" event={"ID":"44b6b6f1-ab7a-4263-8b88-37242d37fac2","Type":"ContainerStarted","Data":"12086b918a65c9744e0d6b2ed60b11906d7e7c562dedcd08511082da86bf4cf2"}
Sep 04 23:52:36.989845 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:36.989731    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m" event={"ID":"44b6b6f1-ab7a-4263-8b88-37242d37fac2","Type":"ContainerStarted","Data":"a46be7c8a1bfb5981086ddec17f0acdfc83a26c71aa20ea2ca092ad8f6bd2782"}
Sep 04 23:52:36.989845 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:36.989760    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"
Sep 04 23:52:37.005725 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:37.005673    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q" podStartSLOduration=2.005657973 podStartE2EDuration="2.005657973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:52:37.002888926 +0000 UTC m=+939.420707235" watchObservedRunningTime="2026-09-04 23:52:37.005657973 +0000 UTC m=+939.423476291"
Sep 04 23:52:37.016892 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:37.016843    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m" podStartSLOduration=2.016828807 podStartE2EDuration="2.016828807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:52:37.016351028 +0000 UTC m=+939.434169347" watchObservedRunningTime="2026-09-04 23:52:37.016828807 +0000 UTC m=+939.434647126"
Sep 04 23:52:40.316871 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:40.316833    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5"]
Sep 04 23:52:40.319407 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:40.319386    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5"
Sep 04 23:52:40.321927 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:40.321896    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 23:52:40.322075 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:40.321962    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k4f64\""
Sep 04 23:52:40.327136 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:40.327101    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5"]
Sep 04 23:52:40.374322 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:40.374292    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fce8f2a-dc5f-4761-bca6-46003acb1937-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5\" (UID: \"3fce8f2a-dc5f-4761-bca6-46003acb1937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5"
Sep 04 23:52:40.475031 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:40.475001    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fce8f2a-dc5f-4761-bca6-46003acb1937-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5\" (UID: \"3fce8f2a-dc5f-4761-bca6-46003acb1937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5"
Sep 04 23:52:40.475584 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:40.475565    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fce8f2a-dc5f-4761-bca6-46003acb1937-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5\" (UID: \"3fce8f2a-dc5f-4761-bca6-46003acb1937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5"
Sep 04 23:52:40.631728 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:40.631638    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5"
Sep 04 23:52:40.907027 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:40.907000    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5"]
Sep 04 23:52:40.908132 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:52:40.908103    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fce8f2a_dc5f_4761_bca6_46003acb1937.slice/crio-324482ef07efea0791f1ed4d47c46f96dd353310ba314fbbddeded52c4abe004 WatchSource:0}: Error finding container 324482ef07efea0791f1ed4d47c46f96dd353310ba314fbbddeded52c4abe004: Status 404 returned error can't find the container with id 324482ef07efea0791f1ed4d47c46f96dd353310ba314fbbddeded52c4abe004
Sep 04 23:52:41.017130 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:41.017102    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5" event={"ID":"3fce8f2a-dc5f-4761-bca6-46003acb1937","Type":"ContainerStarted","Data":"324482ef07efea0791f1ed4d47c46f96dd353310ba314fbbddeded52c4abe004"}
Sep 04 23:52:42.024826 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:42.024796    2557 generic.go:356] "Generic (PLEG): container finished" podID="3fce8f2a-dc5f-4761-bca6-46003acb1937" containerID="6ab110b8cb7cbf2d4c2f0e7cf2f7e7d1f553e0ba32357065143e103aeac5dc4c" exitCode=0
Sep 04 23:52:42.025230 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:42.024853    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5" event={"ID":"3fce8f2a-dc5f-4761-bca6-46003acb1937","Type":"ContainerDied","Data":"6ab110b8cb7cbf2d4c2f0e7cf2f7e7d1f553e0ba32357065143e103aeac5dc4c"}
Sep 04 23:52:42.338451 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:42.338374    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:43.155652 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:43.155631    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5"
Sep 04 23:52:43.297148 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:43.297066    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fce8f2a-dc5f-4761-bca6-46003acb1937-maas-api-service-ca\") pod \"3fce8f2a-dc5f-4761-bca6-46003acb1937\" (UID: \"3fce8f2a-dc5f-4761-bca6-46003acb1937\") "
Sep 04 23:52:43.297457 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:43.297428    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fce8f2a-dc5f-4761-bca6-46003acb1937-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3fce8f2a-dc5f-4761-bca6-46003acb1937" (UID: "3fce8f2a-dc5f-4761-bca6-46003acb1937"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:52:43.398587 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:43.398562    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fce8f2a-dc5f-4761-bca6-46003acb1937-maas-api-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:44.038422 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:44.038396    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5"
Sep 04 23:52:44.038594 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:44.038438    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5" event={"ID":"3fce8f2a-dc5f-4761-bca6-46003acb1937","Type":"ContainerDied","Data":"324482ef07efea0791f1ed4d47c46f96dd353310ba314fbbddeded52c4abe004"}
Sep 04 23:52:44.038594 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:44.038483    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="324482ef07efea0791f1ed4d47c46f96dd353310ba314fbbddeded52c4abe004"
Sep 04 23:52:48.000450 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:48.000397    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"
Sep 04 23:52:48.001452 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:48.001421    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"
Sep 04 23:52:50.013303 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.013267    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:52:50.624450 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.624384    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl" podUID="303b0311-0cb1-4ab8-8af2-9712e81624b9" containerName="main" containerID="cri-o://b4e56eb30bb3286b30a1daad941bcec2a5baabca8b61f1a42544ba96b0ac5748" gracePeriod=45
Sep 04 23:52:50.670716 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:52:50.670679    2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod303b0311_0cb1_4ab8_8af2_9712e81624b9.slice/crio-conmon-b4e56eb30bb3286b30a1daad941bcec2a5baabca8b61f1a42544ba96b0ac5748.scope\": RecentStats: unable to find data in memory cache]"
Sep 04 23:52:50.908665 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.908637    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:50.939051 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.939005    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5"]
Sep 04 23:52:50.941366 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.941340    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8b443-8lxk5"]
Sep 04 23:52:50.968666 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.968643    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-home\") pod \"303b0311-0cb1-4ab8-8af2-9712e81624b9\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") "
Sep 04 23:52:50.968816 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.968684    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303b0311-0cb1-4ab8-8af2-9712e81624b9-tls-certs\") pod \"303b0311-0cb1-4ab8-8af2-9712e81624b9\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") "
Sep 04 23:52:50.968816 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.968749    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-model-cache\") pod \"303b0311-0cb1-4ab8-8af2-9712e81624b9\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") "
Sep 04 23:52:50.968816 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.968773    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-dshm\") pod \"303b0311-0cb1-4ab8-8af2-9712e81624b9\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") "
Sep 04 23:52:50.969012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.968825    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-tmp-dir\") pod \"303b0311-0cb1-4ab8-8af2-9712e81624b9\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") "
Sep 04 23:52:50.969012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.968852    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw6q6\" (UniqueName: \"kubernetes.io/projected/303b0311-0cb1-4ab8-8af2-9712e81624b9-kube-api-access-kw6q6\") pod \"303b0311-0cb1-4ab8-8af2-9712e81624b9\" (UID: \"303b0311-0cb1-4ab8-8af2-9712e81624b9\") "
Sep 04 23:52:50.969012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.968929    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-home" (OuterVolumeSpecName: "home") pod "303b0311-0cb1-4ab8-8af2-9712e81624b9" (UID: "303b0311-0cb1-4ab8-8af2-9712e81624b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:50.969177 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.969079    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "303b0311-0cb1-4ab8-8af2-9712e81624b9" (UID: "303b0311-0cb1-4ab8-8af2-9712e81624b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:50.969239 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.969217    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:50.969239 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.969233    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:50.969426 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.969399    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "303b0311-0cb1-4ab8-8af2-9712e81624b9" (UID: "303b0311-0cb1-4ab8-8af2-9712e81624b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:50.971201 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.971167    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/303b0311-0cb1-4ab8-8af2-9712e81624b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "303b0311-0cb1-4ab8-8af2-9712e81624b9" (UID: "303b0311-0cb1-4ab8-8af2-9712e81624b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:52:50.971437 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.971419    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303b0311-0cb1-4ab8-8af2-9712e81624b9-kube-api-access-kw6q6" (OuterVolumeSpecName: "kube-api-access-kw6q6") pod "303b0311-0cb1-4ab8-8af2-9712e81624b9" (UID: "303b0311-0cb1-4ab8-8af2-9712e81624b9"). InnerVolumeSpecName "kube-api-access-kw6q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:52:50.971547 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:50.971522    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-dshm" (OuterVolumeSpecName: "dshm") pod "303b0311-0cb1-4ab8-8af2-9712e81624b9" (UID: "303b0311-0cb1-4ab8-8af2-9712e81624b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:52:51.069899 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:51.069867    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:51.070285 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:51.069916    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303b0311-0cb1-4ab8-8af2-9712e81624b9-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:51.070285 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:51.069929    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw6q6\" (UniqueName: \"kubernetes.io/projected/303b0311-0cb1-4ab8-8af2-9712e81624b9-kube-api-access-kw6q6\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:51.070285 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:51.069952    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303b0311-0cb1-4ab8-8af2-9712e81624b9-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:52:51.082917 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:51.082880    2557 generic.go:356] "Generic (PLEG): container finished" podID="303b0311-0cb1-4ab8-8af2-9712e81624b9" containerID="b4e56eb30bb3286b30a1daad941bcec2a5baabca8b61f1a42544ba96b0ac5748" exitCode=0
Sep 04 23:52:51.083117 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:51.082958    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"
Sep 04 23:52:51.083117 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:51.082965    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl" event={"ID":"303b0311-0cb1-4ab8-8af2-9712e81624b9","Type":"ContainerDied","Data":"b4e56eb30bb3286b30a1daad941bcec2a5baabca8b61f1a42544ba96b0ac5748"}
Sep 04 23:52:51.083117 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:51.083022    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl" event={"ID":"303b0311-0cb1-4ab8-8af2-9712e81624b9","Type":"ContainerDied","Data":"baaa80e73f59e5fb345557f4f2406e8fd642b9e54bca9ae34ac29ad870c15eff"}
Sep 04 23:52:51.083117 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:51.083048    2557 scope.go:117] "RemoveContainer" containerID="b4e56eb30bb3286b30a1daad941bcec2a5baabca8b61f1a42544ba96b0ac5748"
Sep 04 23:52:51.094719 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:51.094701    2557 scope.go:117] "RemoveContainer" containerID="b4e56eb30bb3286b30a1daad941bcec2a5baabca8b61f1a42544ba96b0ac5748"
Sep 04 23:52:51.095039 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:52:51.095016    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e56eb30bb3286b30a1daad941bcec2a5baabca8b61f1a42544ba96b0ac5748\": container with ID starting with b4e56eb30bb3286b30a1daad941bcec2a5baabca8b61f1a42544ba96b0ac5748 not found: ID does not exist" containerID="b4e56eb30bb3286b30a1daad941bcec2a5baabca8b61f1a42544ba96b0ac5748"
Sep 04 23:52:51.095135 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:51.095045    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e56eb30bb3286b30a1daad941bcec2a5baabca8b61f1a42544ba96b0ac5748"} err="failed to get container status \"b4e56eb30bb3286b30a1daad941bcec2a5baabca8b61f1a42544ba96b0ac5748\": rpc error: code = NotFound desc = could not find container \"b4e56eb30bb3286b30a1daad941bcec2a5baabca8b61f1a42544ba96b0ac5748\": container with ID starting with b4e56eb30bb3286b30a1daad941bcec2a5baabca8b61f1a42544ba96b0ac5748 not found: ID does not exist"
Sep 04 23:52:51.108039 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:51.108015    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"]
Sep 04 23:52:51.113567 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:51.113543    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ffeb50/noten-model-ffeb50-kserve-5c4cf88f59-7hzhl"]
Sep 04 23:52:52.079138 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:52.079102    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="303b0311-0cb1-4ab8-8af2-9712e81624b9" path="/var/lib/kubelet/pods/303b0311-0cb1-4ab8-8af2-9712e81624b9/volumes"
Sep 04 23:52:52.079531 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:52.079440    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fce8f2a-dc5f-4761-bca6-46003acb1937" path="/var/lib/kubelet/pods/3fce8f2a-dc5f-4761-bca6-46003acb1937/volumes"
Sep 04 23:52:54.104049 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:54.104020    2557 generic.go:356] "Generic (PLEG): container finished" podID="aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" containerID="0e00a47fa87ee2b52c0b97bf47ffc4e8ebbd01abf974f03118a7368d2dcb7266" exitCode=0
Sep 04 23:52:54.104456 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:54.104087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl" event={"ID":"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb","Type":"ContainerDied","Data":"0e00a47fa87ee2b52c0b97bf47ffc4e8ebbd01abf974f03118a7368d2dcb7266"}
Sep 04 23:52:54.104513 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:54.104490    2557 scope.go:117] "RemoveContainer" containerID="0e00a47fa87ee2b52c0b97bf47ffc4e8ebbd01abf974f03118a7368d2dcb7266"
Sep 04 23:52:54.105475 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:54.105446    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:52:54.105569 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:54.105513    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:52:54.105569 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:54.105545    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:52:55.117739 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:55.117234    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl" event={"ID":"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb","Type":"ContainerStarted","Data":"67aafeb1cdbd7b8c1eccf4b04318facd22d3be3b8790b3fe2e20863fd99e3713"}
Sep 04 23:52:56.016715 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:56.016679    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:52:56.016918 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:56.016894    2557 patch_prober.go:28] interesting pod/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 04 23:52:56.016985 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:56.016965    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl" podUID="aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 04 23:52:57.017273 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:57.017241    2557 patch_prober.go:28] interesting pod/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 04 23:52:57.017659 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:57.017297    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl" podUID="aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 04 23:52:57.426197 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:57.426112    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"]
Sep 04 23:52:57.426396 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:57.426360    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerName="istio-proxy" containerID="cri-o://a49cf2b360baaf0d1053c84fcdbc5435ab84c20da79131ece8643f77fd2c5770" gracePeriod=30
Sep 04 23:52:58.016809 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:58.016775    2557 patch_prober.go:28] interesting pod/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 04 23:52:58.016978 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:58.016836    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl" podUID="aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 04 23:52:58.614266 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:58.614230    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:52:59.017058 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:59.017020    2557 patch_prober.go:28] interesting pod/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 04 23:52:59.017245 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:59.017082    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl" podUID="aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 04 23:52:59.584602 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:59.584563    2557 patch_prober.go:28] interesting pod/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 04 23:52:59.584797 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:52:59.584643    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 23:53:00.017170 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:00.017131    2557 patch_prober.go:28] interesting pod/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 04 23:53:00.017582 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:00.017208    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl" podUID="aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 04 23:53:01.020792 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:01.020761    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:53:01.021213 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:01.021126    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:53:01.021721 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:01.021701    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:53:02.678237 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.678209    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:53:02.788111 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788084    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-podinfo\") pod \"b149e027-57c4-47fd-a20b-af8c0969fd08\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") "
Sep 04 23:53:02.788248 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788179    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-envoy\") pod \"b149e027-57c4-47fd-a20b-af8c0969fd08\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") "
Sep 04 23:53:02.788248 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788210    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-workload-certs\") pod \"b149e027-57c4-47fd-a20b-af8c0969fd08\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") "
Sep 04 23:53:02.788248 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788238    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-credential-socket\") pod \"b149e027-57c4-47fd-a20b-af8c0969fd08\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") "
Sep 04 23:53:02.788390 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788270    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b149e027-57c4-47fd-a20b-af8c0969fd08-istiod-ca-cert\") pod \"b149e027-57c4-47fd-a20b-af8c0969fd08\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") "
Sep 04 23:53:02.788390 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788311    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s57nh\" (UniqueName: \"kubernetes.io/projected/b149e027-57c4-47fd-a20b-af8c0969fd08-kube-api-access-s57nh\") pod \"b149e027-57c4-47fd-a20b-af8c0969fd08\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") "
Sep 04 23:53:02.788390 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788335    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-workload-socket\") pod \"b149e027-57c4-47fd-a20b-af8c0969fd08\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") "
Sep 04 23:53:02.788390 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788364    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-data\") pod \"b149e027-57c4-47fd-a20b-af8c0969fd08\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") "
Sep 04 23:53:02.788604 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788390    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-token\") pod \"b149e027-57c4-47fd-a20b-af8c0969fd08\" (UID: \"b149e027-57c4-47fd-a20b-af8c0969fd08\") "
Sep 04 23:53:02.788604 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788502    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b149e027-57c4-47fd-a20b-af8c0969fd08" (UID: "b149e027-57c4-47fd-a20b-af8c0969fd08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:53:02.788604 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788537    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b149e027-57c4-47fd-a20b-af8c0969fd08" (UID: "b149e027-57c4-47fd-a20b-af8c0969fd08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:53:02.788757 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788652    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b149e027-57c4-47fd-a20b-af8c0969fd08-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b149e027-57c4-47fd-a20b-af8c0969fd08" (UID: "b149e027-57c4-47fd-a20b-af8c0969fd08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:53:02.788757 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788720    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-data" (OuterVolumeSpecName: "istio-data") pod "b149e027-57c4-47fd-a20b-af8c0969fd08" (UID: "b149e027-57c4-47fd-a20b-af8c0969fd08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:53:02.788866 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788813    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:02.788866 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788834    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:02.788866 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788849    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b149e027-57c4-47fd-a20b-af8c0969fd08-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:02.788866 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788863    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:02.789075 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.788906    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b149e027-57c4-47fd-a20b-af8c0969fd08" (UID: "b149e027-57c4-47fd-a20b-af8c0969fd08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:53:02.790477 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.790449    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b149e027-57c4-47fd-a20b-af8c0969fd08" (UID: "b149e027-57c4-47fd-a20b-af8c0969fd08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:53:02.790477 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.790463    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b149e027-57c4-47fd-a20b-af8c0969fd08" (UID: "b149e027-57c4-47fd-a20b-af8c0969fd08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:53:02.790746 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.790730    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-token" (OuterVolumeSpecName: "istio-token") pod "b149e027-57c4-47fd-a20b-af8c0969fd08" (UID: "b149e027-57c4-47fd-a20b-af8c0969fd08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:53:02.791061 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.791045    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b149e027-57c4-47fd-a20b-af8c0969fd08-kube-api-access-s57nh" (OuterVolumeSpecName: "kube-api-access-s57nh") pod "b149e027-57c4-47fd-a20b-af8c0969fd08" (UID: "b149e027-57c4-47fd-a20b-af8c0969fd08"). InnerVolumeSpecName "kube-api-access-s57nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:53:02.890249 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.890213    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:02.890249 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.890246    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:02.890249 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.890256    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s57nh\" (UniqueName: \"kubernetes.io/projected/b149e027-57c4-47fd-a20b-af8c0969fd08-kube-api-access-s57nh\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:02.890448 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.890265    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b149e027-57c4-47fd-a20b-af8c0969fd08-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:02.890448 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:02.890275    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b149e027-57c4-47fd-a20b-af8c0969fd08-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:03.162395 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:03.162326    2557 generic.go:356] "Generic (PLEG): container finished" podID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerID="a49cf2b360baaf0d1053c84fcdbc5435ab84c20da79131ece8643f77fd2c5770" exitCode=0
Sep 04 23:53:03.162395 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:03.162369    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"
Sep 04 23:53:03.162395 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:03.162374    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" event={"ID":"b149e027-57c4-47fd-a20b-af8c0969fd08","Type":"ContainerDied","Data":"a49cf2b360baaf0d1053c84fcdbc5435ab84c20da79131ece8643f77fd2c5770"}
Sep 04 23:53:03.162596 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:03.162422    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt" event={"ID":"b149e027-57c4-47fd-a20b-af8c0969fd08","Type":"ContainerDied","Data":"33d47c2fa13309e1f33ad67cf8ad35d09a066662b9b8b2438707c3e1f54bf67e"}
Sep 04 23:53:03.162596 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:03.162452    2557 scope.go:117] "RemoveContainer" containerID="a49cf2b360baaf0d1053c84fcdbc5435ab84c20da79131ece8643f77fd2c5770"
Sep 04 23:53:03.172085 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:03.172068    2557 scope.go:117] "RemoveContainer" containerID="98b10948bec8dfaa64bd8b2e7f10c2312a5628a547f636837660d3b648e8c331"
Sep 04 23:53:03.180009 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:03.179992    2557 scope.go:117] "RemoveContainer" containerID="a49cf2b360baaf0d1053c84fcdbc5435ab84c20da79131ece8643f77fd2c5770"
Sep 04 23:53:03.180263 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:53:03.180244    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49cf2b360baaf0d1053c84fcdbc5435ab84c20da79131ece8643f77fd2c5770\": container with ID starting with a49cf2b360baaf0d1053c84fcdbc5435ab84c20da79131ece8643f77fd2c5770 not found: ID does not exist" containerID="a49cf2b360baaf0d1053c84fcdbc5435ab84c20da79131ece8643f77fd2c5770"
Sep 04 23:53:03.180312 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:03.180271    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49cf2b360baaf0d1053c84fcdbc5435ab84c20da79131ece8643f77fd2c5770"} err="failed to get container status \"a49cf2b360baaf0d1053c84fcdbc5435ab84c20da79131ece8643f77fd2c5770\": rpc error: code = NotFound desc = could not find container \"a49cf2b360baaf0d1053c84fcdbc5435ab84c20da79131ece8643f77fd2c5770\": container with ID starting with a49cf2b360baaf0d1053c84fcdbc5435ab84c20da79131ece8643f77fd2c5770 not found: ID does not exist"
Sep 04 23:53:03.180312 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:03.180288    2557 scope.go:117] "RemoveContainer" containerID="98b10948bec8dfaa64bd8b2e7f10c2312a5628a547f636837660d3b648e8c331"
Sep 04 23:53:03.180532 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:53:03.180517    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b10948bec8dfaa64bd8b2e7f10c2312a5628a547f636837660d3b648e8c331\": container with ID starting with 98b10948bec8dfaa64bd8b2e7f10c2312a5628a547f636837660d3b648e8c331 not found: ID does not exist" containerID="98b10948bec8dfaa64bd8b2e7f10c2312a5628a547f636837660d3b648e8c331"
Sep 04 23:53:03.180592 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:03.180535    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b10948bec8dfaa64bd8b2e7f10c2312a5628a547f636837660d3b648e8c331"} err="failed to get container status \"98b10948bec8dfaa64bd8b2e7f10c2312a5628a547f636837660d3b648e8c331\": rpc error: code = NotFound desc = could not find container \"98b10948bec8dfaa64bd8b2e7f10c2312a5628a547f636837660d3b648e8c331\": container with ID starting with 98b10948bec8dfaa64bd8b2e7f10c2312a5628a547f636837660d3b648e8c331 not found: ID does not exist"
Sep 04 23:53:03.184896 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:03.184874    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"]
Sep 04 23:53:03.192099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:03.192076    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d8b443-openshift-default-559b659988-js6mt"]
Sep 04 23:53:04.078250 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:04.078218    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" path="/var/lib/kubelet/pods/b149e027-57c4-47fd-a20b-af8c0969fd08/volumes"
Sep 04 23:53:11.810421 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:11.810390    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:53:14.919130 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:14.919095    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"]
Sep 04 23:53:14.919618 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:14.919318    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7" podUID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerName="istio-proxy" containerID="cri-o://99b2eeece76335e8faeb9c764fbc801d3ce8e0e19c5cd39885054970ba8d5228" gracePeriod=30
Sep 04 23:53:14.947781 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:14.947750    2557 patch_prober.go:28] interesting pod/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 04 23:53:14.947895 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:14.947805    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7" podUID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 23:53:16.676748 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:16.676715    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"]
Sep 04 23:53:16.677234 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:16.676962    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q" podUID="f7189bf6-8406-4fb5-b06c-d8e60f7501d2" containerName="payload-processing" containerID="cri-o://3864a618ecd409961fa6defd4e6b431c132c2f3f220d21c6a585fab269fe53f3" gracePeriod=30
Sep 04 23:53:16.686346 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:16.686319    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"]
Sep 04 23:53:16.686566 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:16.686536    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m" podUID="44b6b6f1-ab7a-4263-8b88-37242d37fac2" containerName="payload-pre-processing" containerID="cri-o://12086b918a65c9744e0d6b2ed60b11906d7e7c562dedcd08511082da86bf4cf2" gracePeriod=30
Sep 04 23:53:16.954109 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:16.954087    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"
Sep 04 23:53:16.956835 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:16.956818    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"
Sep 04 23:53:17.012524 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.011830    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b6b6f1-ab7a-4263-8b88-37242d37fac2-plugins-config-volume\") pod \"44b6b6f1-ab7a-4263-8b88-37242d37fac2\" (UID: \"44b6b6f1-ab7a-4263-8b88-37242d37fac2\") "
Sep 04 23:53:17.012524 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.011924    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2lxf\" (UniqueName: \"kubernetes.io/projected/f7189bf6-8406-4fb5-b06c-d8e60f7501d2-kube-api-access-r2lxf\") pod \"f7189bf6-8406-4fb5-b06c-d8e60f7501d2\" (UID: \"f7189bf6-8406-4fb5-b06c-d8e60f7501d2\") "
Sep 04 23:53:17.012524 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.012006    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc2j4\" (UniqueName: \"kubernetes.io/projected/44b6b6f1-ab7a-4263-8b88-37242d37fac2-kube-api-access-rc2j4\") pod \"44b6b6f1-ab7a-4263-8b88-37242d37fac2\" (UID: \"44b6b6f1-ab7a-4263-8b88-37242d37fac2\") "
Sep 04 23:53:17.012524 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.012033    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7189bf6-8406-4fb5-b06c-d8e60f7501d2-plugins-config-volume\") pod \"f7189bf6-8406-4fb5-b06c-d8e60f7501d2\" (UID: \"f7189bf6-8406-4fb5-b06c-d8e60f7501d2\") "
Sep 04 23:53:17.013115 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.013069    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7189bf6-8406-4fb5-b06c-d8e60f7501d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7189bf6-8406-4fb5-b06c-d8e60f7501d2" (UID: "f7189bf6-8406-4fb5-b06c-d8e60f7501d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:53:17.013481 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.013452    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b6b6f1-ab7a-4263-8b88-37242d37fac2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44b6b6f1-ab7a-4263-8b88-37242d37fac2" (UID: "44b6b6f1-ab7a-4263-8b88-37242d37fac2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:53:17.015905 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.015878    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b6b6f1-ab7a-4263-8b88-37242d37fac2-kube-api-access-rc2j4" (OuterVolumeSpecName: "kube-api-access-rc2j4") pod "44b6b6f1-ab7a-4263-8b88-37242d37fac2" (UID: "44b6b6f1-ab7a-4263-8b88-37242d37fac2"). InnerVolumeSpecName "kube-api-access-rc2j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:53:17.017110 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.017076    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7189bf6-8406-4fb5-b06c-d8e60f7501d2-kube-api-access-r2lxf" (OuterVolumeSpecName: "kube-api-access-r2lxf") pod "f7189bf6-8406-4fb5-b06c-d8e60f7501d2" (UID: "f7189bf6-8406-4fb5-b06c-d8e60f7501d2"). InnerVolumeSpecName "kube-api-access-r2lxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:53:17.113243 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.113206    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc2j4\" (UniqueName: \"kubernetes.io/projected/44b6b6f1-ab7a-4263-8b88-37242d37fac2-kube-api-access-rc2j4\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:17.113243 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.113239    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7189bf6-8406-4fb5-b06c-d8e60f7501d2-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:17.113483 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.113254    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b6b6f1-ab7a-4263-8b88-37242d37fac2-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:17.113483 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.113269    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2lxf\" (UniqueName: \"kubernetes.io/projected/f7189bf6-8406-4fb5-b06c-d8e60f7501d2-kube-api-access-r2lxf\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:17.242692 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.242661    2557 generic.go:356] "Generic (PLEG): container finished" podID="f7189bf6-8406-4fb5-b06c-d8e60f7501d2" containerID="3864a618ecd409961fa6defd4e6b431c132c2f3f220d21c6a585fab269fe53f3" exitCode=0
Sep 04 23:53:17.242844 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.242697    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"
Sep 04 23:53:17.242844 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.242749    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q" event={"ID":"f7189bf6-8406-4fb5-b06c-d8e60f7501d2","Type":"ContainerDied","Data":"3864a618ecd409961fa6defd4e6b431c132c2f3f220d21c6a585fab269fe53f3"}
Sep 04 23:53:17.242844 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.242803    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q" event={"ID":"f7189bf6-8406-4fb5-b06c-d8e60f7501d2","Type":"ContainerDied","Data":"cfd3d65860098cb9ec3e2d7b13509a4957b8f0d0811dd3e544195c40dbbee91b"}
Sep 04 23:53:17.242844 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.242830    2557 scope.go:117] "RemoveContainer" containerID="3864a618ecd409961fa6defd4e6b431c132c2f3f220d21c6a585fab269fe53f3"
Sep 04 23:53:17.244149 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.244124    2557 generic.go:356] "Generic (PLEG): container finished" podID="44b6b6f1-ab7a-4263-8b88-37242d37fac2" containerID="12086b918a65c9744e0d6b2ed60b11906d7e7c562dedcd08511082da86bf4cf2" exitCode=0
Sep 04 23:53:17.244255 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.244172    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"
Sep 04 23:53:17.244255 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.244190    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m" event={"ID":"44b6b6f1-ab7a-4263-8b88-37242d37fac2","Type":"ContainerDied","Data":"12086b918a65c9744e0d6b2ed60b11906d7e7c562dedcd08511082da86bf4cf2"}
Sep 04 23:53:17.244255 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.244237    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m" event={"ID":"44b6b6f1-ab7a-4263-8b88-37242d37fac2","Type":"ContainerDied","Data":"a46be7c8a1bfb5981086ddec17f0acdfc83a26c71aa20ea2ca092ad8f6bd2782"}
Sep 04 23:53:17.253692 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.253584    2557 scope.go:117] "RemoveContainer" containerID="3864a618ecd409961fa6defd4e6b431c132c2f3f220d21c6a585fab269fe53f3"
Sep 04 23:53:17.254001 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:53:17.253974    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3864a618ecd409961fa6defd4e6b431c132c2f3f220d21c6a585fab269fe53f3\": container with ID starting with 3864a618ecd409961fa6defd4e6b431c132c2f3f220d21c6a585fab269fe53f3 not found: ID does not exist" containerID="3864a618ecd409961fa6defd4e6b431c132c2f3f220d21c6a585fab269fe53f3"
Sep 04 23:53:17.254090 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.254011    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3864a618ecd409961fa6defd4e6b431c132c2f3f220d21c6a585fab269fe53f3"} err="failed to get container status \"3864a618ecd409961fa6defd4e6b431c132c2f3f220d21c6a585fab269fe53f3\": rpc error: code = NotFound desc = could not find container \"3864a618ecd409961fa6defd4e6b431c132c2f3f220d21c6a585fab269fe53f3\": container with ID starting with 3864a618ecd409961fa6defd4e6b431c132c2f3f220d21c6a585fab269fe53f3 not found: ID does not exist"
Sep 04 23:53:17.254090 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.254040    2557 scope.go:117] "RemoveContainer" containerID="12086b918a65c9744e0d6b2ed60b11906d7e7c562dedcd08511082da86bf4cf2"
Sep 04 23:53:17.261851 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.261835    2557 scope.go:117] "RemoveContainer" containerID="12086b918a65c9744e0d6b2ed60b11906d7e7c562dedcd08511082da86bf4cf2"
Sep 04 23:53:17.262132 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:53:17.262113    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12086b918a65c9744e0d6b2ed60b11906d7e7c562dedcd08511082da86bf4cf2\": container with ID starting with 12086b918a65c9744e0d6b2ed60b11906d7e7c562dedcd08511082da86bf4cf2 not found: ID does not exist" containerID="12086b918a65c9744e0d6b2ed60b11906d7e7c562dedcd08511082da86bf4cf2"
Sep 04 23:53:17.262203 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.262139    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12086b918a65c9744e0d6b2ed60b11906d7e7c562dedcd08511082da86bf4cf2"} err="failed to get container status \"12086b918a65c9744e0d6b2ed60b11906d7e7c562dedcd08511082da86bf4cf2\": rpc error: code = NotFound desc = could not find container \"12086b918a65c9744e0d6b2ed60b11906d7e7c562dedcd08511082da86bf4cf2\": container with ID starting with 12086b918a65c9744e0d6b2ed60b11906d7e7c562dedcd08511082da86bf4cf2 not found: ID does not exist"
Sep 04 23:53:17.267141 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.267120    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"]
Sep 04 23:53:17.271012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.270990    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f4c99d38-7599899f57-67j7q"]
Sep 04 23:53:17.287207 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.287176    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"]
Sep 04 23:53:17.287313 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:17.287224    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f4c99d38-84bd4dd558-tgf9m"]
Sep 04 23:53:18.078770 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:18.078737    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44b6b6f1-ab7a-4263-8b88-37242d37fac2" path="/var/lib/kubelet/pods/44b6b6f1-ab7a-4263-8b88-37242d37fac2/volumes"
Sep 04 23:53:18.079289 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:18.079254    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7189bf6-8406-4fb5-b06c-d8e60f7501d2" path="/var/lib/kubelet/pods/f7189bf6-8406-4fb5-b06c-d8e60f7501d2/volumes"
Sep 04 23:53:19.920099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:19.920065    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:53:20.161731 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.161706    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:53:20.239163 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239126    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-credential-socket\") pod \"b44c0b19-e574-41e7-96c4-9fab38801f45\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") "
Sep 04 23:53:20.239356 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239170    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-data\") pod \"b44c0b19-e574-41e7-96c4-9fab38801f45\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") "
Sep 04 23:53:20.239356 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239201    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-podinfo\") pod \"b44c0b19-e574-41e7-96c4-9fab38801f45\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") "
Sep 04 23:53:20.239356 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239234    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqpz4\" (UniqueName: \"kubernetes.io/projected/b44c0b19-e574-41e7-96c4-9fab38801f45-kube-api-access-hqpz4\") pod \"b44c0b19-e574-41e7-96c4-9fab38801f45\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") "
Sep 04 23:53:20.239356 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239289    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-workload-socket\") pod \"b44c0b19-e574-41e7-96c4-9fab38801f45\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") "
Sep 04 23:53:20.239589 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239412    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-token\") pod \"b44c0b19-e574-41e7-96c4-9fab38801f45\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") "
Sep 04 23:53:20.239589 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239415    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b44c0b19-e574-41e7-96c4-9fab38801f45" (UID: "b44c0b19-e574-41e7-96c4-9fab38801f45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:53:20.239589 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239430    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-data" (OuterVolumeSpecName: "istio-data") pod "b44c0b19-e574-41e7-96c4-9fab38801f45" (UID: "b44c0b19-e574-41e7-96c4-9fab38801f45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:53:20.239589 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239454    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-envoy\") pod \"b44c0b19-e574-41e7-96c4-9fab38801f45\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") "
Sep 04 23:53:20.239589 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239490    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44c0b19-e574-41e7-96c4-9fab38801f45-istiod-ca-cert\") pod \"b44c0b19-e574-41e7-96c4-9fab38801f45\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") "
Sep 04 23:53:20.239589 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239520    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-workload-certs\") pod \"b44c0b19-e574-41e7-96c4-9fab38801f45\" (UID: \"b44c0b19-e574-41e7-96c4-9fab38801f45\") "
Sep 04 23:53:20.239589 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239571    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b44c0b19-e574-41e7-96c4-9fab38801f45" (UID: "b44c0b19-e574-41e7-96c4-9fab38801f45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:53:20.239978 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239855    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b44c0b19-e574-41e7-96c4-9fab38801f45" (UID: "b44c0b19-e574-41e7-96c4-9fab38801f45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:53:20.239978 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239879    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:20.239978 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239903    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:20.239978 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239917    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:20.239978 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.239884    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44c0b19-e574-41e7-96c4-9fab38801f45-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b44c0b19-e574-41e7-96c4-9fab38801f45" (UID: "b44c0b19-e574-41e7-96c4-9fab38801f45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:53:20.241509 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.241480    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-token" (OuterVolumeSpecName: "istio-token") pod "b44c0b19-e574-41e7-96c4-9fab38801f45" (UID: "b44c0b19-e574-41e7-96c4-9fab38801f45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:53:20.241640 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.241588    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b44c0b19-e574-41e7-96c4-9fab38801f45" (UID: "b44c0b19-e574-41e7-96c4-9fab38801f45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:53:20.241801 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.241778    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b44c0b19-e574-41e7-96c4-9fab38801f45" (UID: "b44c0b19-e574-41e7-96c4-9fab38801f45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:53:20.241874 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.241844    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44c0b19-e574-41e7-96c4-9fab38801f45-kube-api-access-hqpz4" (OuterVolumeSpecName: "kube-api-access-hqpz4") pod "b44c0b19-e574-41e7-96c4-9fab38801f45" (UID: "b44c0b19-e574-41e7-96c4-9fab38801f45"). InnerVolumeSpecName "kube-api-access-hqpz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:53:20.264481 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.264450    2557 generic.go:356] "Generic (PLEG): container finished" podID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerID="99b2eeece76335e8faeb9c764fbc801d3ce8e0e19c5cd39885054970ba8d5228" exitCode=0
Sep 04 23:53:20.264576 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.264509    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"
Sep 04 23:53:20.264576 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.264539    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7" event={"ID":"b44c0b19-e574-41e7-96c4-9fab38801f45","Type":"ContainerDied","Data":"99b2eeece76335e8faeb9c764fbc801d3ce8e0e19c5cd39885054970ba8d5228"}
Sep 04 23:53:20.264648 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.264591    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7" event={"ID":"b44c0b19-e574-41e7-96c4-9fab38801f45","Type":"ContainerDied","Data":"9e5016f71a81600554b6fa8297c176d04a7337487a83f21ad93e664596480498"}
Sep 04 23:53:20.264648 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.264617    2557 scope.go:117] "RemoveContainer" containerID="99b2eeece76335e8faeb9c764fbc801d3ce8e0e19c5cd39885054970ba8d5228"
Sep 04 23:53:20.274325 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.274307    2557 scope.go:117] "RemoveContainer" containerID="7dcb281a389a1f1d7f81e42bb8d720a94cd6b2ea9f501f5fff335482f35b65ab"
Sep 04 23:53:20.282448 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.282433    2557 scope.go:117] "RemoveContainer" containerID="99b2eeece76335e8faeb9c764fbc801d3ce8e0e19c5cd39885054970ba8d5228"
Sep 04 23:53:20.282684 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:53:20.282667    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b2eeece76335e8faeb9c764fbc801d3ce8e0e19c5cd39885054970ba8d5228\": container with ID starting with 99b2eeece76335e8faeb9c764fbc801d3ce8e0e19c5cd39885054970ba8d5228 not found: ID does not exist" containerID="99b2eeece76335e8faeb9c764fbc801d3ce8e0e19c5cd39885054970ba8d5228"
Sep 04 23:53:20.282738 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.282691    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b2eeece76335e8faeb9c764fbc801d3ce8e0e19c5cd39885054970ba8d5228"} err="failed to get container status \"99b2eeece76335e8faeb9c764fbc801d3ce8e0e19c5cd39885054970ba8d5228\": rpc error: code = NotFound desc = could not find container \"99b2eeece76335e8faeb9c764fbc801d3ce8e0e19c5cd39885054970ba8d5228\": container with ID starting with 99b2eeece76335e8faeb9c764fbc801d3ce8e0e19c5cd39885054970ba8d5228 not found: ID does not exist"
Sep 04 23:53:20.282738 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.282708    2557 scope.go:117] "RemoveContainer" containerID="7dcb281a389a1f1d7f81e42bb8d720a94cd6b2ea9f501f5fff335482f35b65ab"
Sep 04 23:53:20.282989 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:53:20.282970    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dcb281a389a1f1d7f81e42bb8d720a94cd6b2ea9f501f5fff335482f35b65ab\": container with ID starting with 7dcb281a389a1f1d7f81e42bb8d720a94cd6b2ea9f501f5fff335482f35b65ab not found: ID does not exist" containerID="7dcb281a389a1f1d7f81e42bb8d720a94cd6b2ea9f501f5fff335482f35b65ab"
Sep 04 23:53:20.283062 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.282994    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dcb281a389a1f1d7f81e42bb8d720a94cd6b2ea9f501f5fff335482f35b65ab"} err="failed to get container status \"7dcb281a389a1f1d7f81e42bb8d720a94cd6b2ea9f501f5fff335482f35b65ab\": rpc error: code = NotFound desc = could not find container \"7dcb281a389a1f1d7f81e42bb8d720a94cd6b2ea9f501f5fff335482f35b65ab\": container with ID starting with 7dcb281a389a1f1d7f81e42bb8d720a94cd6b2ea9f501f5fff335482f35b65ab not found: ID does not exist"
Sep 04 23:53:20.288867 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.288848    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"]
Sep 04 23:53:20.292278 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.292233    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-724e1f-openshift-default-58cd7f5c78-pbhd7"]
Sep 04 23:53:20.340982 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.340927    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:20.340982 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.340973    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqpz4\" (UniqueName: \"kubernetes.io/projected/b44c0b19-e574-41e7-96c4-9fab38801f45-kube-api-access-hqpz4\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:20.341147 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.340988    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:20.341147 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.341000    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:20.341147 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.341011    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44c0b19-e574-41e7-96c4-9fab38801f45-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:20.341147 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.341022    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44c0b19-e574-41e7-96c4-9fab38801f45-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:20.481095 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481065    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"]
Sep 04 23:53:20.481397 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481385    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fce8f2a-dc5f-4761-bca6-46003acb1937" containerName="revoke-keys"
Sep 04 23:53:20.481446 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481399    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fce8f2a-dc5f-4761-bca6-46003acb1937" containerName="revoke-keys"
Sep 04 23:53:20.481446 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481414    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerName="istio-proxy"
Sep 04 23:53:20.481446 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481420    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerName="istio-proxy"
Sep 04 23:53:20.481446 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481434    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7189bf6-8406-4fb5-b06c-d8e60f7501d2" containerName="payload-processing"
Sep 04 23:53:20.481446 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481439    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7189bf6-8406-4fb5-b06c-d8e60f7501d2" containerName="payload-processing"
Sep 04 23:53:20.481446 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481446    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="303b0311-0cb1-4ab8-8af2-9712e81624b9" containerName="main"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481452    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="303b0311-0cb1-4ab8-8af2-9712e81624b9" containerName="main"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481459    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerName="istio-proxy"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481464    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerName="istio-proxy"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481471    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerName="istio-proxy"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481477    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerName="istio-proxy"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481484    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b6b6f1-ab7a-4263-8b88-37242d37fac2" containerName="payload-pre-processing"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481489    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b6b6f1-ab7a-4263-8b88-37242d37fac2" containerName="payload-pre-processing"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481498    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerName="istio-proxy"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481503    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerName="istio-proxy"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481553    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerName="istio-proxy"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481559    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b149e027-57c4-47fd-a20b-af8c0969fd08" containerName="istio-proxy"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481566    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerName="istio-proxy"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481571    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44c0b19-e574-41e7-96c4-9fab38801f45" containerName="istio-proxy"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481577    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fce8f2a-dc5f-4761-bca6-46003acb1937" containerName="revoke-keys"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481582    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="303b0311-0cb1-4ab8-8af2-9712e81624b9" containerName="main"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481588    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b6b6f1-ab7a-4263-8b88-37242d37fac2" containerName="payload-pre-processing"
Sep 04 23:53:20.481627 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.481594    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7189bf6-8406-4fb5-b06c-d8e60f7501d2" containerName="payload-processing"
Sep 04 23:53:20.486057 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.486041    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"
Sep 04 23:53:20.488854 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.488825    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-5245c1\""
Sep 04 23:53:20.489021 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.488904    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-5245c1-dockercfg-jr5sq\""
Sep 04 23:53:20.492913 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.492858    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"]
Sep 04 23:53:20.542081 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.542052    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jsg9\" (UniqueName: \"kubernetes.io/projected/c1ded87b-823c-46c2-be19-513efbcc60b3-kube-api-access-9jsg9\") pod \"payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6\" (UID: \"c1ded87b-823c-46c2-be19-513efbcc60b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"
Sep 04 23:53:20.542188 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.542089    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ded87b-823c-46c2-be19-513efbcc60b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6\" (UID: \"c1ded87b-823c-46c2-be19-513efbcc60b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"
Sep 04 23:53:20.643085 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.643052    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ded87b-823c-46c2-be19-513efbcc60b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6\" (UID: \"c1ded87b-823c-46c2-be19-513efbcc60b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"
Sep 04 23:53:20.643256 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.643169    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jsg9\" (UniqueName: \"kubernetes.io/projected/c1ded87b-823c-46c2-be19-513efbcc60b3-kube-api-access-9jsg9\") pod \"payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6\" (UID: \"c1ded87b-823c-46c2-be19-513efbcc60b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"
Sep 04 23:53:20.643861 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.643837    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ded87b-823c-46c2-be19-513efbcc60b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6\" (UID: \"c1ded87b-823c-46c2-be19-513efbcc60b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"
Sep 04 23:53:20.650836 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.650811    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jsg9\" (UniqueName: \"kubernetes.io/projected/c1ded87b-823c-46c2-be19-513efbcc60b3-kube-api-access-9jsg9\") pod \"payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6\" (UID: \"c1ded87b-823c-46c2-be19-513efbcc60b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"
Sep 04 23:53:20.797084 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:20.797020    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"
Sep 04 23:53:21.066245 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:21.066218    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"]
Sep 04 23:53:21.067468 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:53:21.067442    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ded87b_823c_46c2_be19_513efbcc60b3.slice/crio-71c2aaa862a6c8664de28e443e1627312f9916aaabd1acd7cd8245a538b014f7 WatchSource:0}: Error finding container 71c2aaa862a6c8664de28e443e1627312f9916aaabd1acd7cd8245a538b014f7: Status 404 returned error can't find the container with id 71c2aaa862a6c8664de28e443e1627312f9916aaabd1acd7cd8245a538b014f7
Sep 04 23:53:21.272319 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:21.272282    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6" event={"ID":"c1ded87b-823c-46c2-be19-513efbcc60b3","Type":"ContainerStarted","Data":"a3204f0e6513b2010f7438fafa6c12ccbc8bfe75d338d6ceeacb70815fb2f449"}
Sep 04 23:53:21.272495 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:21.272374    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6" event={"ID":"c1ded87b-823c-46c2-be19-513efbcc60b3","Type":"ContainerStarted","Data":"71c2aaa862a6c8664de28e443e1627312f9916aaabd1acd7cd8245a538b014f7"}
Sep 04 23:53:21.272495 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:21.272402    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"
Sep 04 23:53:21.287671 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:21.287625    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6" podStartSLOduration=1.287608993 podStartE2EDuration="1.287608993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:53:21.286206204 +0000 UTC m=+983.704024524" watchObservedRunningTime="2026-09-04 23:53:21.287608993 +0000 UTC m=+983.705427311"
Sep 04 23:53:22.078678 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:22.078642    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b44c0b19-e574-41e7-96c4-9fab38801f45" path="/var/lib/kubelet/pods/b44c0b19-e574-41e7-96c4-9fab38801f45/volumes"
Sep 04 23:53:22.906611 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:22.906581    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"]
Sep 04 23:53:22.906843 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:22.906818    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl" podUID="aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" containerName="istio-proxy" containerID="cri-o://67aafeb1cdbd7b8c1eccf4b04318facd22d3be3b8790b3fe2e20863fd99e3713" gracePeriod=30
Sep 04 23:53:28.161549 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.161524    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:53:28.308298 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.308264    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-credential-socket\") pod \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") "
Sep 04 23:53:28.308474 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.308320    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhlzf\" (UniqueName: \"kubernetes.io/projected/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-kube-api-access-qhlzf\") pod \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") "
Sep 04 23:53:28.308474 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.308344    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-workload-socket\") pod \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") "
Sep 04 23:53:28.308474 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.308360    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-workload-certs\") pod \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") "
Sep 04 23:53:28.308474 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.308374    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istiod-ca-cert\") pod \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") "
Sep 04 23:53:28.308474 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.308428    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-podinfo\") pod \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") "
Sep 04 23:53:28.308474 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.308469    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-token\") pod \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") "
Sep 04 23:53:28.308798 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.308501    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-envoy\") pod \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") "
Sep 04 23:53:28.308798 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.308536    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-data\") pod \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\" (UID: \"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb\") "
Sep 04 23:53:28.308798 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.308626    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" (UID: "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:53:28.309048 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.308787    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" (UID: "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:53:28.309048 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.308578    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" (UID: "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:53:28.309048 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.308972    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" (UID: "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:53:28.309048 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.308988    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" (UID: "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:53:28.310990 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.310958    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-kube-api-access-qhlzf" (OuterVolumeSpecName: "kube-api-access-qhlzf") pod "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" (UID: "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb"). InnerVolumeSpecName "kube-api-access-qhlzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:53:28.311159 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.311128    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" (UID: "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:53:28.311281 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.311244    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" (UID: "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:53:28.311281 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.311256    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" (UID: "aa3220ae-ffa8-4ba2-95fd-7127685cf6fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:53:28.315338 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.315312    2557 generic.go:356] "Generic (PLEG): container finished" podID="aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" containerID="67aafeb1cdbd7b8c1eccf4b04318facd22d3be3b8790b3fe2e20863fd99e3713" exitCode=0
Sep 04 23:53:28.315448 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.315381    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"
Sep 04 23:53:28.315448 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.315388    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl" event={"ID":"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb","Type":"ContainerDied","Data":"67aafeb1cdbd7b8c1eccf4b04318facd22d3be3b8790b3fe2e20863fd99e3713"}
Sep 04 23:53:28.315565 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.315464    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl" event={"ID":"aa3220ae-ffa8-4ba2-95fd-7127685cf6fb","Type":"ContainerDied","Data":"4f09149a254cbddd91226a20b7b359932c660e901c3c5cbc64b34ac45fc4307f"}
Sep 04 23:53:28.315565 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.315487    2557 scope.go:117] "RemoveContainer" containerID="67aafeb1cdbd7b8c1eccf4b04318facd22d3be3b8790b3fe2e20863fd99e3713"
Sep 04 23:53:28.329207 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.329189    2557 scope.go:117] "RemoveContainer" containerID="0e00a47fa87ee2b52c0b97bf47ffc4e8ebbd01abf974f03118a7368d2dcb7266"
Sep 04 23:53:28.336818 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.336797    2557 scope.go:117] "RemoveContainer" containerID="67aafeb1cdbd7b8c1eccf4b04318facd22d3be3b8790b3fe2e20863fd99e3713"
Sep 04 23:53:28.337224 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:53:28.337196    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67aafeb1cdbd7b8c1eccf4b04318facd22d3be3b8790b3fe2e20863fd99e3713\": container with ID starting with 67aafeb1cdbd7b8c1eccf4b04318facd22d3be3b8790b3fe2e20863fd99e3713 not found: ID does not exist" containerID="67aafeb1cdbd7b8c1eccf4b04318facd22d3be3b8790b3fe2e20863fd99e3713"
Sep 04 23:53:28.337328 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.337225    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67aafeb1cdbd7b8c1eccf4b04318facd22d3be3b8790b3fe2e20863fd99e3713"} err="failed to get container status \"67aafeb1cdbd7b8c1eccf4b04318facd22d3be3b8790b3fe2e20863fd99e3713\": rpc error: code = NotFound desc = could not find container \"67aafeb1cdbd7b8c1eccf4b04318facd22d3be3b8790b3fe2e20863fd99e3713\": container with ID starting with 67aafeb1cdbd7b8c1eccf4b04318facd22d3be3b8790b3fe2e20863fd99e3713 not found: ID does not exist"
Sep 04 23:53:28.337328 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.337245    2557 scope.go:117] "RemoveContainer" containerID="0e00a47fa87ee2b52c0b97bf47ffc4e8ebbd01abf974f03118a7368d2dcb7266"
Sep 04 23:53:28.337670 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:53:28.337641    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e00a47fa87ee2b52c0b97bf47ffc4e8ebbd01abf974f03118a7368d2dcb7266\": container with ID starting with 0e00a47fa87ee2b52c0b97bf47ffc4e8ebbd01abf974f03118a7368d2dcb7266 not found: ID does not exist" containerID="0e00a47fa87ee2b52c0b97bf47ffc4e8ebbd01abf974f03118a7368d2dcb7266"
Sep 04 23:53:28.337771 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.337696    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e00a47fa87ee2b52c0b97bf47ffc4e8ebbd01abf974f03118a7368d2dcb7266"} err="failed to get container status \"0e00a47fa87ee2b52c0b97bf47ffc4e8ebbd01abf974f03118a7368d2dcb7266\": rpc error: code = NotFound desc = could not find container \"0e00a47fa87ee2b52c0b97bf47ffc4e8ebbd01abf974f03118a7368d2dcb7266\": container with ID starting with 0e00a47fa87ee2b52c0b97bf47ffc4e8ebbd01abf974f03118a7368d2dcb7266 not found: ID does not exist"
Sep 04 23:53:28.340164 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.340145    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"]
Sep 04 23:53:28.343282 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.343261    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f4c99d38-openshift-default-5c4b6955b4-5gfwl"]
Sep 04 23:53:28.409299 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.409278    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhlzf\" (UniqueName: \"kubernetes.io/projected/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-kube-api-access-qhlzf\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:28.409299 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.409299    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:28.409418 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.409309    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:28.409418 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.409318    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:28.409418 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.409326    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:28.409418 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.409333    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:28.409418 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.409340    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:28.409418 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.409347    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:28.409418 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:28.409354    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:53:30.079127 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:30.079091    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" path="/var/lib/kubelet/pods/aa3220ae-ffa8-4ba2-95fd-7127685cf6fb/volumes"
Sep 04 23:53:30.611241 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:30.611207    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:53:32.279643 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:32.279614    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"
Sep 04 23:53:36.124300 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.124266    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"]
Sep 04 23:53:36.124710 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.124635    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" containerName="istio-proxy"
Sep 04 23:53:36.124710 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.124648    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" containerName="istio-proxy"
Sep 04 23:53:36.124710 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.124667    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" containerName="istio-proxy"
Sep 04 23:53:36.124710 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.124672    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" containerName="istio-proxy"
Sep 04 23:53:36.124865 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.124756    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" containerName="istio-proxy"
Sep 04 23:53:36.124865 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.124765    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa3220ae-ffa8-4ba2-95fd-7127685cf6fb" containerName="istio-proxy"
Sep 04 23:53:36.129349 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.129331    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.131823 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.131805    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-e35897-openshift-default-dockercfg-h5tzr\""
Sep 04 23:53:36.140653 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.140629    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"]
Sep 04 23:53:36.274050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.274007    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-workload-certs\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.274231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.274054    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-credential-socket\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.274231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.274115    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421752b2-6564-4051-bc67-c34348a27f6a-istio-podinfo\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.274231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.274143    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421752b2-6564-4051-bc67-c34348a27f6a-istiod-ca-cert\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.274231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.274171    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-istio-envoy\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.274231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.274212    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztjkw\" (UniqueName: \"kubernetes.io/projected/421752b2-6564-4051-bc67-c34348a27f6a-kube-api-access-ztjkw\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.274434 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.274245    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421752b2-6564-4051-bc67-c34348a27f6a-istio-token\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.274434 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.274281    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-workload-socket\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.274434 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.274303    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-istio-data\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.375166 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.375081    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-istio-envoy\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.375166 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.375136    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztjkw\" (UniqueName: \"kubernetes.io/projected/421752b2-6564-4051-bc67-c34348a27f6a-kube-api-access-ztjkw\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.375388 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.375164    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421752b2-6564-4051-bc67-c34348a27f6a-istio-token\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.375388 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.375193    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-workload-socket\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.375388 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.375219    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-istio-data\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.375388 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.375265    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-workload-certs\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.375388 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.375299    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-credential-socket\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.375388 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.375356    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421752b2-6564-4051-bc67-c34348a27f6a-istio-podinfo\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.375697 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.375404    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421752b2-6564-4051-bc67-c34348a27f6a-istiod-ca-cert\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.375697 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.375578    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-workload-socket\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.375697 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.375686    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-istio-data\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.375857 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.375706    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-workload-certs\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.375857 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.375787    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-credential-socket\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.376146 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.376125    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421752b2-6564-4051-bc67-c34348a27f6a-istiod-ca-cert\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.377366 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.377347    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-istio-envoy\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.377920 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.377899    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421752b2-6564-4051-bc67-c34348a27f6a-istio-podinfo\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.382580 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.382553    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421752b2-6564-4051-bc67-c34348a27f6a-istio-token\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.382848 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.382832    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztjkw\" (UniqueName: \"kubernetes.io/projected/421752b2-6564-4051-bc67-c34348a27f6a-kube-api-access-ztjkw\") pod \"e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") " pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.441929 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.441904    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:36.716835 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.716799    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"]
Sep 04 23:53:36.717633 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:53:36.717606    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421752b2_6564_4051_bc67_c34348a27f6a.slice/crio-38203ac6bcdaaad3fa0629d9fddf1724d4320d65ababba7c65993f7b5d513890 WatchSource:0}: Error finding container 38203ac6bcdaaad3fa0629d9fddf1724d4320d65ababba7c65993f7b5d513890: Status 404 returned error can't find the container with id 38203ac6bcdaaad3fa0629d9fddf1724d4320d65ababba7c65993f7b5d513890
Sep 04 23:53:36.719831 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.719797    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:53:36.719919 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.719858    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:53:36.719919 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:36.719885    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:53:37.366845 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:37.366806    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" event={"ID":"421752b2-6564-4051-bc67-c34348a27f6a","Type":"ContainerStarted","Data":"a0c8043ba841694944b49d0f9e9ebe8de127b5ddfe0f125719eaca1715c8a18c"}
Sep 04 23:53:37.367214 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:37.366866    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" event={"ID":"421752b2-6564-4051-bc67-c34348a27f6a","Type":"ContainerStarted","Data":"38203ac6bcdaaad3fa0629d9fddf1724d4320d65ababba7c65993f7b5d513890"}
Sep 04 23:53:37.387561 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:37.387510    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" podStartSLOduration=1.387489094 podStartE2EDuration="1.387489094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:53:37.385484858 +0000 UTC m=+999.803303189" watchObservedRunningTime="2026-09-04 23:53:37.387489094 +0000 UTC m=+999.805307414"
Sep 04 23:53:37.443012 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:37.442983    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:37.444547 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:37.444524    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 04 23:53:37.444643 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:37.444580    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" podUID="421752b2-6564-4051-bc67-c34348a27f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 04 23:53:38.445861 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:38.445833    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:39.376692 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:39.376667    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:39.377617 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:39.377594    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:41.132929 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:41.132890    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"]
Sep 04 23:53:41.136725 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:41.136701    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"
Sep 04 23:53:41.139285 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:41.139259    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-e35897\""
Sep 04 23:53:41.147751 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:41.147728    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"]
Sep 04 23:53:41.219688 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:41.219658    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b20687-9123-40e3-95ab-6caa7aa51c28-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965\" (UID: \"78b20687-9123-40e3-95ab-6caa7aa51c28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"
Sep 04 23:53:41.219822 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:41.219696    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l7nb\" (UniqueName: \"kubernetes.io/projected/78b20687-9123-40e3-95ab-6caa7aa51c28-kube-api-access-8l7nb\") pod \"payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965\" (UID: \"78b20687-9123-40e3-95ab-6caa7aa51c28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"
Sep 04 23:53:41.320700 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:41.320664    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b20687-9123-40e3-95ab-6caa7aa51c28-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965\" (UID: \"78b20687-9123-40e3-95ab-6caa7aa51c28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"
Sep 04 23:53:41.320700 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:41.320704    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l7nb\" (UniqueName: \"kubernetes.io/projected/78b20687-9123-40e3-95ab-6caa7aa51c28-kube-api-access-8l7nb\") pod \"payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965\" (UID: \"78b20687-9123-40e3-95ab-6caa7aa51c28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"
Sep 04 23:53:41.321428 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:41.321402    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b20687-9123-40e3-95ab-6caa7aa51c28-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965\" (UID: \"78b20687-9123-40e3-95ab-6caa7aa51c28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"
Sep 04 23:53:41.329089 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:41.329067    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l7nb\" (UniqueName: \"kubernetes.io/projected/78b20687-9123-40e3-95ab-6caa7aa51c28-kube-api-access-8l7nb\") pod \"payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965\" (UID: \"78b20687-9123-40e3-95ab-6caa7aa51c28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"
Sep 04 23:53:41.449703 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:41.449681    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"
Sep 04 23:53:41.719180 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:41.719158    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"]
Sep 04 23:53:41.720551 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:53:41.720526    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b20687_9123_40e3_95ab_6caa7aa51c28.slice/crio-7936d608fe8e36f2dc9b95aff154926a38bd0d979b2880519d0bdc11b863b91b WatchSource:0}: Error finding container 7936d608fe8e36f2dc9b95aff154926a38bd0d979b2880519d0bdc11b863b91b: Status 404 returned error can't find the container with id 7936d608fe8e36f2dc9b95aff154926a38bd0d979b2880519d0bdc11b863b91b
Sep 04 23:53:42.395106 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:42.395071    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965" event={"ID":"78b20687-9123-40e3-95ab-6caa7aa51c28","Type":"ContainerStarted","Data":"888e5aadf5e5851b5e8804fbbd21045c664604a1ca8694d9c1acc2fbcae73f28"}
Sep 04 23:53:42.395456 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:42.395131    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965" event={"ID":"78b20687-9123-40e3-95ab-6caa7aa51c28","Type":"ContainerStarted","Data":"7936d608fe8e36f2dc9b95aff154926a38bd0d979b2880519d0bdc11b863b91b"}
Sep 04 23:53:42.395456 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:42.395168    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"
Sep 04 23:53:42.412903 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:42.412863    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965" podStartSLOduration=1.412851749 podStartE2EDuration="1.412851749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:53:42.411743011 +0000 UTC m=+1004.829561330" watchObservedRunningTime="2026-09-04 23:53:42.412851749 +0000 UTC m=+1004.830670068"
Sep 04 23:53:44.260915 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.260883    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"]
Sep 04 23:53:44.264631 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.264613    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.266959 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.266910    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-88828d-openshift-default-dockercfg-tmr4b\""
Sep 04 23:53:44.275800 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.275778    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"]
Sep 04 23:53:44.449043 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.448974    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-podinfo\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.449242 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.449066    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istiod-ca-cert\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.449242 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.449096    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-workload-certs\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.449242 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.449140    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-envoy\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.449242 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.449168    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-277fz\" (UniqueName: \"kubernetes.io/projected/11d6c9bd-fae5-412a-b7fc-c3319e634b29-kube-api-access-277fz\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.449242 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.449222    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-token\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.449512 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.449256    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-workload-socket\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.449512 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.449307    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-credential-socket\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.449512 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.449334    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-data\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.550126 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.550043    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-envoy\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.550126 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.550077    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-277fz\" (UniqueName: \"kubernetes.io/projected/11d6c9bd-fae5-412a-b7fc-c3319e634b29-kube-api-access-277fz\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.550126 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.550112    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-token\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.550364 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.550145    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-workload-socket\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.550364 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.550195    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-credential-socket\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.550364 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.550218    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-data\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.550364 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.550283    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-podinfo\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.550364 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.550319    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istiod-ca-cert\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.550647 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.550481    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-workload-socket\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.550647 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.550483    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-workload-certs\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.550647 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.550625    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-credential-socket\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.550810 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.550689    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-data\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.550865 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.550802    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-workload-certs\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.551030 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.551012    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istiod-ca-cert\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.552390 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.552366    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-envoy\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.552612 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.552594    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-podinfo\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.557350 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.557330    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-277fz\" (UniqueName: \"kubernetes.io/projected/11d6c9bd-fae5-412a-b7fc-c3319e634b29-kube-api-access-277fz\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.557426 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.557369    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-token\") pod \"e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") " pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.576479 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.576460    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:44.854041 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.854015    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"]
Sep 04 23:53:44.855695 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:53:44.855663    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d6c9bd_fae5_412a_b7fc_c3319e634b29.slice/crio-3fb91b7841fe86446fd9930a84d019fc008e7e92c2a2492dc543adfba648a050 WatchSource:0}: Error finding container 3fb91b7841fe86446fd9930a84d019fc008e7e92c2a2492dc543adfba648a050: Status 404 returned error can't find the container with id 3fb91b7841fe86446fd9930a84d019fc008e7e92c2a2492dc543adfba648a050
Sep 04 23:53:44.859978 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.858800    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:53:44.859978 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.859018    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:53:44.859978 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:44.859117    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:53:45.121685 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:45.121603    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:53:45.413818 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:45.413729    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm" event={"ID":"11d6c9bd-fae5-412a-b7fc-c3319e634b29","Type":"ContainerStarted","Data":"8017483328ccd8cd29163954e4174dc0bf12d46629aeda6873bc734be0b4ae44"}
Sep 04 23:53:45.413818 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:45.413792    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm" event={"ID":"11d6c9bd-fae5-412a-b7fc-c3319e634b29","Type":"ContainerStarted","Data":"3fb91b7841fe86446fd9930a84d019fc008e7e92c2a2492dc543adfba648a050"}
Sep 04 23:53:45.434199 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:45.434156    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm" podStartSLOduration=1.4341413 podStartE2EDuration="1.4341413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:53:45.431266558 +0000 UTC m=+1007.849084876" watchObservedRunningTime="2026-09-04 23:53:45.4341413 +0000 UTC m=+1007.851959617"
Sep 04 23:53:45.577196 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:45.577168    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:45.581970 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:45.581919    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:46.419152 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:46.419119    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:46.420118 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:46.420099    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:53:49.014370 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:49.014335    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"]
Sep 04 23:53:49.018066 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:49.018047    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"
Sep 04 23:53:49.022312 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:49.022293    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-88828d-dockercfg-bw92f\""
Sep 04 23:53:49.022548 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:49.022532    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-88828d\""
Sep 04 23:53:49.029671 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:49.029647    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"]
Sep 04 23:53:49.113691 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:49.113533    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jg84\" (UniqueName: \"kubernetes.io/projected/d8bdac91-7d4e-48a9-a384-1eaf25625fd3-kube-api-access-4jg84\") pod \"payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt\" (UID: \"d8bdac91-7d4e-48a9-a384-1eaf25625fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"
Sep 04 23:53:49.114165 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:49.114136    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8bdac91-7d4e-48a9-a384-1eaf25625fd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt\" (UID: \"d8bdac91-7d4e-48a9-a384-1eaf25625fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"
Sep 04 23:53:49.216178 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:49.216137    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8bdac91-7d4e-48a9-a384-1eaf25625fd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt\" (UID: \"d8bdac91-7d4e-48a9-a384-1eaf25625fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"
Sep 04 23:53:49.216379 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:49.216231    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jg84\" (UniqueName: \"kubernetes.io/projected/d8bdac91-7d4e-48a9-a384-1eaf25625fd3-kube-api-access-4jg84\") pod \"payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt\" (UID: \"d8bdac91-7d4e-48a9-a384-1eaf25625fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"
Sep 04 23:53:49.216896 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:49.216870    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8bdac91-7d4e-48a9-a384-1eaf25625fd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt\" (UID: \"d8bdac91-7d4e-48a9-a384-1eaf25625fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"
Sep 04 23:53:49.225197 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:49.225178    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jg84\" (UniqueName: \"kubernetes.io/projected/d8bdac91-7d4e-48a9-a384-1eaf25625fd3-kube-api-access-4jg84\") pod \"payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt\" (UID: \"d8bdac91-7d4e-48a9-a384-1eaf25625fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"
Sep 04 23:53:49.329930 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:49.329848    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"
Sep 04 23:53:49.618039 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:49.618006    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"]
Sep 04 23:53:49.619145 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:53:49.619116    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8bdac91_7d4e_48a9_a384_1eaf25625fd3.slice/crio-1abf3a7efbd430429753c05983cf40014af19df5ac0fbc67117995f90e401b7a WatchSource:0}: Error finding container 1abf3a7efbd430429753c05983cf40014af19df5ac0fbc67117995f90e401b7a: Status 404 returned error can't find the container with id 1abf3a7efbd430429753c05983cf40014af19df5ac0fbc67117995f90e401b7a
Sep 04 23:53:50.444307 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:50.444257    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt" event={"ID":"d8bdac91-7d4e-48a9-a384-1eaf25625fd3","Type":"ContainerStarted","Data":"7c81445d524f2ca60760ebcc30c1530477d78378b24d2253e2c6d2431324f48b"}
Sep 04 23:53:50.444708 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:50.444317    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt" event={"ID":"d8bdac91-7d4e-48a9-a384-1eaf25625fd3","Type":"ContainerStarted","Data":"1abf3a7efbd430429753c05983cf40014af19df5ac0fbc67117995f90e401b7a"}
Sep 04 23:53:50.444708 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:50.444424    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"
Sep 04 23:53:50.461786 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:50.461741    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt" podStartSLOduration=2.461726195 podStartE2EDuration="2.461726195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:53:50.458921702 +0000 UTC m=+1012.876740019" watchObservedRunningTime="2026-09-04 23:53:50.461726195 +0000 UTC m=+1012.879544513"
Sep 04 23:53:53.403771 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:53.402881    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"
Sep 04 23:53:55.419101 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:55.419062    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 04 23:53:55.419609 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:55.419124    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" podUID="421752b2-6564-4051-bc67-c34348a27f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 04 23:53:56.485984 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:56.485929    2557 generic.go:356] "Generic (PLEG): container finished" podID="421752b2-6564-4051-bc67-c34348a27f6a" containerID="a0c8043ba841694944b49d0f9e9ebe8de127b5ddfe0f125719eaca1715c8a18c" exitCode=0
Sep 04 23:53:56.486791 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:56.486610    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" event={"ID":"421752b2-6564-4051-bc67-c34348a27f6a","Type":"ContainerDied","Data":"a0c8043ba841694944b49d0f9e9ebe8de127b5ddfe0f125719eaca1715c8a18c"}
Sep 04 23:53:56.487236 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:56.487207    2557 scope.go:117] "RemoveContainer" containerID="a0c8043ba841694944b49d0f9e9ebe8de127b5ddfe0f125719eaca1715c8a18c"
Sep 04 23:53:56.488902 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:56.488578    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:53:56.488902 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:56.488668    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:53:56.488902 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:56.488715    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:53:57.499825 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:57.499784    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" event={"ID":"421752b2-6564-4051-bc67-c34348a27f6a","Type":"ContainerStarted","Data":"ac365edca550b65f466d9636f510ef4a99a3594f591c87499569d2f44a984718"}
Sep 04 23:53:58.442679 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:58.442595    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:53:58.442831 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:58.442693    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 04 23:53:58.442831 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:58.442737    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" podUID="421752b2-6564-4051-bc67-c34348a27f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 04 23:53:59.211834 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:59.211795    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:53:59.442409 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:59.442327    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 04 23:53:59.442409 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:53:59.442391    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" podUID="421752b2-6564-4051-bc67-c34348a27f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 04 23:54:00.443121 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:00.443086    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 04 23:54:00.443602 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:00.443155    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" podUID="421752b2-6564-4051-bc67-c34348a27f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 04 23:54:01.442470 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:01.442432    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 04 23:54:01.442645 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:01.442503    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" podUID="421752b2-6564-4051-bc67-c34348a27f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 04 23:54:01.452274 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:01.452250    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"
Sep 04 23:54:02.442968 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:02.442922    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 04 23:54:02.443174 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:02.442991    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" podUID="421752b2-6564-4051-bc67-c34348a27f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 04 23:54:02.461082 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:02.461052    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 04 23:54:02.461438 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:02.461105    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm" podUID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 04 23:54:03.446214 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:03.446186    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:54:03.446505 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:03.446483    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:54:03.447036 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:03.447015    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 04 23:54:03.542014 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:03.541984    2557 generic.go:356] "Generic (PLEG): container finished" podID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerID="8017483328ccd8cd29163954e4174dc0bf12d46629aeda6873bc734be0b4ae44" exitCode=0
Sep 04 23:54:03.542368 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:03.542070    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm" event={"ID":"11d6c9bd-fae5-412a-b7fc-c3319e634b29","Type":"ContainerDied","Data":"8017483328ccd8cd29163954e4174dc0bf12d46629aeda6873bc734be0b4ae44"}
Sep 04 23:54:03.542600 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:03.542586    2557 scope.go:117] "RemoveContainer" containerID="8017483328ccd8cd29163954e4174dc0bf12d46629aeda6873bc734be0b4ae44"
Sep 04 23:54:03.543600 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:03.543571    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:54:03.543726 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:03.543649    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:54:03.543726 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:03.543702    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:54:04.549090 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.549053    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm" event={"ID":"11d6c9bd-fae5-412a-b7fc-c3319e634b29","Type":"ContainerStarted","Data":"b000c5065bb98a9cce59e7b175e5b9cf55d1f0bc0164a34d8eff82f6f172764e"}
Sep 04 23:54:04.577610 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.577582    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:54:04.577778 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.577757    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 04 23:54:04.577858 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.577812    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm" podUID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 04 23:54:04.778162 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.778126    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"]
Sep 04 23:54:04.791727 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.791698    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.794200 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.793399    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"]
Sep 04 23:54:04.794616 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.794593    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-f7eb8e-openshift-default-dockercfg-fnqbc\""
Sep 04 23:54:04.878735 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.878704    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-data\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.878900 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.878742    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-workload-socket\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.878900 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.878782    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-credential-socket\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.878900 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.878801    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istiod-ca-cert\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.878900 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.878864    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4flw4\" (UniqueName: \"kubernetes.io/projected/d46a512a-b1f4-4d41-98c1-a044b37afd9c-kube-api-access-4flw4\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.878900 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.878890    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-workload-certs\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.879171 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.878930    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-envoy\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.879171 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.879000    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-podinfo\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.879171 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.879025    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-token\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.980065 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.980024    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-workload-certs\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.980241 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.980078    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-envoy\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.980241 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.980130    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-podinfo\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.980241 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.980154    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-token\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.980241 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.980214    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-data\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.980507 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.980242    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-workload-socket\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.980507 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.980307    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-credential-socket\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.980507 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.980333    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istiod-ca-cert\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.980507 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.980371    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4flw4\" (UniqueName: \"kubernetes.io/projected/d46a512a-b1f4-4d41-98c1-a044b37afd9c-kube-api-access-4flw4\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.980733 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.980501    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-workload-certs\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.981037 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.981008    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-data\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.981870 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.981841    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-credential-socket\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.982178 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.982152    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-workload-socket\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.982907 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.982878    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istiod-ca-cert\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.984965 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.983188    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-envoy\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.985435 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.985409    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-podinfo\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.990195 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.990167    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-token\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:04.996580 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:04.996537    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4flw4\" (UniqueName: \"kubernetes.io/projected/d46a512a-b1f4-4d41-98c1-a044b37afd9c-kube-api-access-4flw4\") pod \"e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") " pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:05.113485 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:05.113395    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:05.390218 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:05.390128    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"]
Sep 04 23:54:05.409972 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:05.409855    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:54:05.409972 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:05.409960    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:54:05.410111 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:05.410011    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:54:05.558265 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:05.558222    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq" event={"ID":"d46a512a-b1f4-4d41-98c1-a044b37afd9c","Type":"ContainerStarted","Data":"552bdfe59124b21ae546fcd29c33f700428c4a4c16c088b4a45944f49aa74d79"}
Sep 04 23:54:05.558726 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:05.558287    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq" event={"ID":"d46a512a-b1f4-4d41-98c1-a044b37afd9c","Type":"ContainerStarted","Data":"66c2e86a8802383705509dd33745f805b09b601470ad08fcb879e0b38cd7028f"}
Sep 04 23:54:05.577535 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:05.577500    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 04 23:54:05.577660 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:05.577550    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm" podUID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 04 23:54:06.114273 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:06.114233    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:06.116021 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:06.115995    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 04 23:54:06.116135 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:06.116049    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq" podUID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 04 23:54:06.577438 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:06.577407    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 04 23:54:06.577799 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:06.577454    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm" podUID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 04 23:54:07.118507 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:07.118471    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:07.137189 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:07.137138    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq" podStartSLOduration=3.137123851 podStartE2EDuration="3.137123851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:54:05.576161921 +0000 UTC m=+1027.993980240" watchObservedRunningTime="2026-09-04 23:54:07.137123851 +0000 UTC m=+1029.554942169"
Sep 04 23:54:07.569722 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:07.569693    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:07.570755 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:07.570738    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:07.576923 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:07.576890    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 04 23:54:07.577117 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:07.576982    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm" podUID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 04 23:54:08.582993 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:08.578927    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 04 23:54:08.583575 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:08.583542    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm" podUID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 04 23:54:09.593556 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:09.593522    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:54:09.594150 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:09.594121    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:54:09.595571 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:09.595550    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 04 23:54:15.323497 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:15.323462    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:54:23.611077 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:23.611040    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 04 23:54:23.611549 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:23.611116    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq" podUID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 04 23:54:25.676985 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:25.676949    2557 generic.go:356] "Generic (PLEG): container finished" podID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerID="552bdfe59124b21ae546fcd29c33f700428c4a4c16c088b4a45944f49aa74d79" exitCode=0
Sep 04 23:54:25.676985 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:25.676969    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq" event={"ID":"d46a512a-b1f4-4d41-98c1-a044b37afd9c","Type":"ContainerDied","Data":"552bdfe59124b21ae546fcd29c33f700428c4a4c16c088b4a45944f49aa74d79"}
Sep 04 23:54:25.677432 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:25.677410    2557 scope.go:117] "RemoveContainer" containerID="552bdfe59124b21ae546fcd29c33f700428c4a4c16c088b4a45944f49aa74d79"
Sep 04 23:54:25.678658 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:25.678618    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:54:25.678759 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:25.678700    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:54:25.678759 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:25.678731    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:54:26.114751 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.114719    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"]
Sep 04 23:54:26.117795 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.117772    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.120321 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.120296    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-5245c1\"/\"openshift-service-ca.crt\""
Sep 04 23:54:26.121423 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.121406    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5245c1\"/\"default-dockercfg-bdmqt\""
Sep 04 23:54:26.121528 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.121449    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-5245c1\"/\"kube-root-ca.crt\""
Sep 04 23:54:26.121753 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.121734    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5245c1\"/\"auth-test-model-5245c1-kserve-self-signed-certs\""
Sep 04 23:54:26.126024 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.125917    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"]
Sep 04 23:54:26.197846 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.197795    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-dshm\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.198065 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.197863    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnkth\" (UniqueName: \"kubernetes.io/projected/fa3917c2-789c-410b-a52b-19ef94b2e7b4-kube-api-access-bnkth\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.198065 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.197900    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-tmp-dir\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.198065 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.197955    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-model-cache\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.198261 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.198077    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-home\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.198261 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.198101    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3917c2-789c-410b-a52b-19ef94b2e7b4-tls-certs\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.299445 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.299395    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-dshm\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.299667 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.299464    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnkth\" (UniqueName: \"kubernetes.io/projected/fa3917c2-789c-410b-a52b-19ef94b2e7b4-kube-api-access-bnkth\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.299667 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.299496    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-tmp-dir\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.299667 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.299533    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-model-cache\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.299667 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.299611    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-home\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.299667 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.299635    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3917c2-789c-410b-a52b-19ef94b2e7b4-tls-certs\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.300599 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.300570    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-tmp-dir\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.301154 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.301131    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-model-cache\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.301265 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.301238    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-home\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.306742 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.306709    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3917c2-789c-410b-a52b-19ef94b2e7b4-tls-certs\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.307053 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.307028    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-dshm\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.309229 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.309207    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnkth\" (UniqueName: \"kubernetes.io/projected/fa3917c2-789c-410b-a52b-19ef94b2e7b4-kube-api-access-bnkth\") pod \"auth-test-model-5245c1-kserve-d7d496fd7-bpc5q\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") " pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.430116 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.430009    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:26.688417 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.688327    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq" event={"ID":"d46a512a-b1f4-4d41-98c1-a044b37afd9c","Type":"ContainerStarted","Data":"91a3483df2dabd764bc52c7060f1189cd1a6a2a1517fe63678c4333c0deb6dc7"}
Sep 04 23:54:26.717664 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:26.717628    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"]
Sep 04 23:54:26.719028 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:54:26.719002    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa3917c2_789c_410b_a52b_19ef94b2e7b4.slice/crio-082204accdd35beabcaf397cab2f27ce4e80958e7b6fe3622079253a8f99f79a WatchSource:0}: Error finding container 082204accdd35beabcaf397cab2f27ce4e80958e7b6fe3622079253a8f99f79a: Status 404 returned error can't find the container with id 082204accdd35beabcaf397cab2f27ce4e80958e7b6fe3622079253a8f99f79a
Sep 04 23:54:27.113797 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:27.113755    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:27.113999 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:27.113974    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 04 23:54:27.114076 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:27.114031    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq" podUID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 04 23:54:27.700715 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:27.700672    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q" event={"ID":"fa3917c2-789c-410b-a52b-19ef94b2e7b4","Type":"ContainerStarted","Data":"5e51968085c85f8ae167f4414b50e84a881cce7ff83ab9f68e038cf098240f66"}
Sep 04 23:54:27.701221 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:27.700725    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q" event={"ID":"fa3917c2-789c-410b-a52b-19ef94b2e7b4","Type":"ContainerStarted","Data":"082204accdd35beabcaf397cab2f27ce4e80958e7b6fe3622079253a8f99f79a"}
Sep 04 23:54:27.718188 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:27.718144    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q" podStartSLOduration=1.718129891 podStartE2EDuration="1.718129891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:54:27.715285266 +0000 UTC m=+1050.133103584" watchObservedRunningTime="2026-09-04 23:54:27.718129891 +0000 UTC m=+1050.135948208"
Sep 04 23:54:28.114516 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:28.114437    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 04 23:54:28.114516 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:28.114485    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq" podUID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 04 23:54:29.113919 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:29.113880    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 04 23:54:29.114356 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:29.113972    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq" podUID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 04 23:54:29.618545 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:29.618513    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:54:30.114251 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:30.114216    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 04 23:54:30.114644 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:30.114271    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq" podUID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 04 23:54:31.114278 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:31.114244    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 04 23:54:31.114642 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:31.114294    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq" podUID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 04 23:54:32.117816 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:32.117788    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:32.118285 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:32.118072    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:32.118682 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:32.118666    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 04 23:54:36.431014 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:36.430978    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:36.431014 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:36.431018    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:36.443859 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:36.443836    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:36.774084 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:36.774059    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:54:41.657233 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.657196    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"]
Sep 04 23:54:41.660305 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.660284    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.662866 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.662835    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ede0daf1-openshift-default-dockercfg-kblj6\""
Sep 04 23:54:41.670979 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.670951    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"]
Sep 04 23:54:41.756974 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.756675    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67hkl\" (UniqueName: \"kubernetes.io/projected/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-kube-api-access-67hkl\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.756974 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.756781    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istiod-ca-cert\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.756974 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.756819    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-workload-socket\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.756974 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.756856    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-podinfo\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.756974 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.756904    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-workload-certs\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.756974 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.756971    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-token\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.757430 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.757017    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-credential-socket\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.757430 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.757066    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-envoy\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.757430 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.757099    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-data\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.858156 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.858128    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-podinfo\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.858342 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.858163    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-workload-certs\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.858342 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.858188    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-token\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.858342 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.858212    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-credential-socket\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.858342 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.858240    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-envoy\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.858552 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.858436    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-data\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.858552 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.858519    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-workload-certs\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.858552 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.858534    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hkl\" (UniqueName: \"kubernetes.io/projected/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-kube-api-access-67hkl\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.858703 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.858562    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-credential-socket\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.858703 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.858620    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istiod-ca-cert\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.858703 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.858657    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-workload-socket\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.858840 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.858717    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-data\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.858984 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.858962    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-workload-socket\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.859259 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.859242    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istiod-ca-cert\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.860473 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.860442    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-envoy\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.860706 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.860689    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-podinfo\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.865882 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.865855    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-token\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.866026 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.865859    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67hkl\" (UniqueName: \"kubernetes.io/projected/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-kube-api-access-67hkl\") pod \"e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") " pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:41.972031 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:41.972010    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:42.245707 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:42.245677    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"]
Sep 04 23:54:42.247525 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:54:42.247501    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6fee78_9e89_40fd_a47a_4a97cb7957c9.slice/crio-54fdff4284ea9e54cd4c59a867c53f8a42273ed1724a0a72fc6757ec67814883 WatchSource:0}: Error finding container 54fdff4284ea9e54cd4c59a867c53f8a42273ed1724a0a72fc6757ec67814883: Status 404 returned error can't find the container with id 54fdff4284ea9e54cd4c59a867c53f8a42273ed1724a0a72fc6757ec67814883
Sep 04 23:54:42.249818 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:42.249784    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:54:42.249891 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:42.249855    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:54:42.249962 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:42.249892    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:54:42.798710 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:42.798669    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc" event={"ID":"4f6fee78-9e89-40fd-a47a-4a97cb7957c9","Type":"ContainerStarted","Data":"440114ab621a3483849021854dfcd6803e74e5c01d151eaa5f96fa9d018fd055"}
Sep 04 23:54:42.799144 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:42.798739    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc" event={"ID":"4f6fee78-9e89-40fd-a47a-4a97cb7957c9","Type":"ContainerStarted","Data":"54fdff4284ea9e54cd4c59a867c53f8a42273ed1724a0a72fc6757ec67814883"}
Sep 04 23:54:42.820890 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:42.820842    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc" podStartSLOduration=1.820826012 podStartE2EDuration="1.820826012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:54:42.817554888 +0000 UTC m=+1065.235373205" watchObservedRunningTime="2026-09-04 23:54:42.820826012 +0000 UTC m=+1065.238644329"
Sep 04 23:54:42.972436 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:42.972401    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:43.978078 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:43.978048    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:44.809568 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:44.809530    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:44.810339 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:44.810318    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:54:46.447779 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:46.447748    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"]
Sep 04 23:54:46.450268 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:46.450249    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"
Sep 04 23:54:46.452859 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:46.452836    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ede0daf1\""
Sep 04 23:54:46.453003 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:46.452862    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ede0daf1-dockercfg-ngsld\""
Sep 04 23:54:46.460661 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:46.460623    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"]
Sep 04 23:54:46.499640 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:46.499611    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612fda7e-6bf2-4179-b07e-c8cf64fb34e4-plugins-config-volume\") pod \"payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458\" (UID: \"612fda7e-6bf2-4179-b07e-c8cf64fb34e4\") " pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"
Sep 04 23:54:46.499743 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:46.499657    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxwk\" (UniqueName: \"kubernetes.io/projected/612fda7e-6bf2-4179-b07e-c8cf64fb34e4-kube-api-access-vhxwk\") pod \"payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458\" (UID: \"612fda7e-6bf2-4179-b07e-c8cf64fb34e4\") " pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"
Sep 04 23:54:46.600363 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:46.600330    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612fda7e-6bf2-4179-b07e-c8cf64fb34e4-plugins-config-volume\") pod \"payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458\" (UID: \"612fda7e-6bf2-4179-b07e-c8cf64fb34e4\") " pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"
Sep 04 23:54:46.600535 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:46.600374    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxwk\" (UniqueName: \"kubernetes.io/projected/612fda7e-6bf2-4179-b07e-c8cf64fb34e4-kube-api-access-vhxwk\") pod \"payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458\" (UID: \"612fda7e-6bf2-4179-b07e-c8cf64fb34e4\") " pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"
Sep 04 23:54:46.601041 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:46.601022    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612fda7e-6bf2-4179-b07e-c8cf64fb34e4-plugins-config-volume\") pod \"payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458\" (UID: \"612fda7e-6bf2-4179-b07e-c8cf64fb34e4\") " pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"
Sep 04 23:54:46.610769 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:46.610746    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxwk\" (UniqueName: \"kubernetes.io/projected/612fda7e-6bf2-4179-b07e-c8cf64fb34e4-kube-api-access-vhxwk\") pod \"payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458\" (UID: \"612fda7e-6bf2-4179-b07e-c8cf64fb34e4\") " pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"
Sep 04 23:54:46.761833 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:46.761767    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"
Sep 04 23:54:47.031463 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:47.031432    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"]
Sep 04 23:54:47.032839 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:54:47.032813    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612fda7e_6bf2_4179_b07e_c8cf64fb34e4.slice/crio-cb036d03662bb0d7a8a5ded33395e69009ae4e20a4d941761747e5880af8ca0f WatchSource:0}: Error finding container cb036d03662bb0d7a8a5ded33395e69009ae4e20a4d941761747e5880af8ca0f: Status 404 returned error can't find the container with id cb036d03662bb0d7a8a5ded33395e69009ae4e20a4d941761747e5880af8ca0f
Sep 04 23:54:47.832039 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:47.832006    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458" event={"ID":"612fda7e-6bf2-4179-b07e-c8cf64fb34e4","Type":"ContainerStarted","Data":"9d4997285c36dead45935f91ac7d16ed6b84ad653308a46844088413c7f93e02"}
Sep 04 23:54:47.832426 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:47.832062    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458" event={"ID":"612fda7e-6bf2-4179-b07e-c8cf64fb34e4","Type":"ContainerStarted","Data":"cb036d03662bb0d7a8a5ded33395e69009ae4e20a4d941761747e5880af8ca0f"}
Sep 04 23:54:47.832426 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:47.832085    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"
Sep 04 23:54:47.852236 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:47.852185    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458" podStartSLOduration=1.852165545 podStartE2EDuration="1.852165545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:54:47.848157667 +0000 UTC m=+1070.265975985" watchObservedRunningTime="2026-09-04 23:54:47.852165545 +0000 UTC m=+1070.269983868"
Sep 04 23:54:52.716321 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:52.716237    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:54:58.168558 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:58.168524    2557 scope.go:117] "RemoveContainer" containerID="437ec0d4c3a2f96b586a1458f7084d32f0eef8664735d645f49e466d6a0eacda"
Sep 04 23:54:58.839973 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:54:58.839919    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"
Sep 04 23:55:02.946647 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:02.946619    2557 generic.go:356] "Generic (PLEG): container finished" podID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" containerID="440114ab621a3483849021854dfcd6803e74e5c01d151eaa5f96fa9d018fd055" exitCode=0
Sep 04 23:55:02.947026 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:02.946673    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc" event={"ID":"4f6fee78-9e89-40fd-a47a-4a97cb7957c9","Type":"ContainerDied","Data":"440114ab621a3483849021854dfcd6803e74e5c01d151eaa5f96fa9d018fd055"}
Sep 04 23:55:02.947098 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:02.947073    2557 scope.go:117] "RemoveContainer" containerID="440114ab621a3483849021854dfcd6803e74e5c01d151eaa5f96fa9d018fd055"
Sep 04 23:55:02.948242 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:02.948210    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:55:02.948294 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:02.948283    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:55:02.948333 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:02.948314    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:55:03.954149 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:03.954117    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc" event={"ID":"4f6fee78-9e89-40fd-a47a-4a97cb7957c9","Type":"ContainerStarted","Data":"3fc126dc7bc5282808dae217db8090551460b98fd17e9bca97ddc7584cee1fda"}
Sep 04 23:55:03.972111 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:03.972087    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:55:04.973403 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:04.973370    2557 patch_prober.go:28] interesting pod/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 04 23:55:04.973802 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:04.973424    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc" podUID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 04 23:55:05.972899 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:05.972865    2557 patch_prober.go:28] interesting pod/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 04 23:55:05.973099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:05.972919    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc" podUID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 04 23:55:06.974145 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:06.974103    2557 patch_prober.go:28] interesting pod/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 04 23:55:06.974562 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:06.974163    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc" podUID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 04 23:55:07.972403 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:07.972367    2557 patch_prober.go:28] interesting pod/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 04 23:55:07.972582 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:07.972419    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc" podUID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 04 23:55:08.972326 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:08.972290    2557 patch_prober.go:28] interesting pod/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 04 23:55:08.972738 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:08.972346    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc" podUID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 04 23:55:09.975717 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:09.975685    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:55:09.976130 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:09.975925    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:55:09.976536 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:09.976518    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:55:10.514611 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:10.514574    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:55:33.125520 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:33.125488    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:55:52.213911 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:52.213875    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:55:58.201353 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:58.201324    2557 scope.go:117] "RemoveContainer" containerID="a15260ea07f10c4b0c7e90d378d4dfc60a2196abe1a5f98b6671b0d91c60d5fd"
Sep 04 23:55:59.226952 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:59.226906    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6"]
Sep 04 23:55:59.230147 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:59.230129    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6"
Sep 04 23:55:59.232897 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:59.232869    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 23:55:59.233081 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:59.232873    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k4f64\""
Sep 04 23:55:59.238391 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:59.238067    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6"]
Sep 04 23:55:59.358191 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:59.358158    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5666a8-5dca-4af3-b900-8ea30ebd9e5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6\" (UID: \"bd5666a8-5dca-4af3-b900-8ea30ebd9e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6"
Sep 04 23:55:59.459309 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:59.459278    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5666a8-5dca-4af3-b900-8ea30ebd9e5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6\" (UID: \"bd5666a8-5dca-4af3-b900-8ea30ebd9e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6"
Sep 04 23:55:59.459825 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:59.459808    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5666a8-5dca-4af3-b900-8ea30ebd9e5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6\" (UID: \"bd5666a8-5dca-4af3-b900-8ea30ebd9e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6"
Sep 04 23:55:59.542953 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:59.542875    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6"
Sep 04 23:55:59.814400 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:55:59.814373    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6"]
Sep 04 23:55:59.815666 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:55:59.815636    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5666a8_5dca_4af3_b900_8ea30ebd9e5b.slice/crio-8f37797284699fbdd1e163ba3c6e3ee99cc5091e3047d01fa8421451a35879d3 WatchSource:0}: Error finding container 8f37797284699fbdd1e163ba3c6e3ee99cc5091e3047d01fa8421451a35879d3: Status 404 returned error can't find the container with id 8f37797284699fbdd1e163ba3c6e3ee99cc5091e3047d01fa8421451a35879d3
Sep 04 23:56:00.304303 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:00.304270    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6" event={"ID":"bd5666a8-5dca-4af3-b900-8ea30ebd9e5b","Type":"ContainerStarted","Data":"2599cf4843a96684e83d69bcf19cd9965024b1c2190a3f61c1ba72a11362c280"}
Sep 04 23:56:00.304683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:00.304330    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6" event={"ID":"bd5666a8-5dca-4af3-b900-8ea30ebd9e5b","Type":"ContainerStarted","Data":"8f37797284699fbdd1e163ba3c6e3ee99cc5091e3047d01fa8421451a35879d3"}
Sep 04 23:56:00.325850 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:00.323027    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6" podStartSLOduration=1.32300861 podStartE2EDuration="1.32300861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:56:00.31946839 +0000 UTC m=+1142.737286707" watchObservedRunningTime="2026-09-04 23:56:00.32300861 +0000 UTC m=+1142.740826928"
Sep 04 23:56:01.312165 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:01.312130    2557 generic.go:356] "Generic (PLEG): container finished" podID="bd5666a8-5dca-4af3-b900-8ea30ebd9e5b" containerID="2599cf4843a96684e83d69bcf19cd9965024b1c2190a3f61c1ba72a11362c280" exitCode=0
Sep 04 23:56:01.312568 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:01.312214    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6" event={"ID":"bd5666a8-5dca-4af3-b900-8ea30ebd9e5b","Type":"ContainerDied","Data":"2599cf4843a96684e83d69bcf19cd9965024b1c2190a3f61c1ba72a11362c280"}
Sep 04 23:56:02.450543 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:02.450520    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6"
Sep 04 23:56:02.480311 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:02.480283    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5666a8-5dca-4af3-b900-8ea30ebd9e5b-maas-api-service-ca\") pod \"bd5666a8-5dca-4af3-b900-8ea30ebd9e5b\" (UID: \"bd5666a8-5dca-4af3-b900-8ea30ebd9e5b\") "
Sep 04 23:56:02.480600 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:02.480574    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5666a8-5dca-4af3-b900-8ea30ebd9e5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd5666a8-5dca-4af3-b900-8ea30ebd9e5b" (UID: "bd5666a8-5dca-4af3-b900-8ea30ebd9e5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:56:02.580962 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:02.580878    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5666a8-5dca-4af3-b900-8ea30ebd9e5b-maas-api-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:56:03.324539 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:03.324511    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6"
Sep 04 23:56:03.324701 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:03.324544    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6" event={"ID":"bd5666a8-5dca-4af3-b900-8ea30ebd9e5b","Type":"ContainerDied","Data":"8f37797284699fbdd1e163ba3c6e3ee99cc5091e3047d01fa8421451a35879d3"}
Sep 04 23:56:03.324701 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:03.324586    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f37797284699fbdd1e163ba3c6e3ee99cc5091e3047d01fa8421451a35879d3"
Sep 04 23:56:06.918552 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:06.918513    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:56:14.845477 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:14.845440    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"]
Sep 04 23:56:14.846319 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:14.846258    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458" podUID="612fda7e-6bf2-4179-b07e-c8cf64fb34e4" containerName="payload-processing" containerID="cri-o://9d4997285c36dead45935f91ac7d16ed6b84ad653308a46844088413c7f93e02" gracePeriod=30
Sep 04 23:56:15.025338 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.025261    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6"]
Sep 04 23:56:15.028960 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.028926    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ede0daf1-dcbn6"]
Sep 04 23:56:15.119681 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.119653    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"
Sep 04 23:56:15.186287 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.186257    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhxwk\" (UniqueName: \"kubernetes.io/projected/612fda7e-6bf2-4179-b07e-c8cf64fb34e4-kube-api-access-vhxwk\") pod \"612fda7e-6bf2-4179-b07e-c8cf64fb34e4\" (UID: \"612fda7e-6bf2-4179-b07e-c8cf64fb34e4\") "
Sep 04 23:56:15.186287 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.186299    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612fda7e-6bf2-4179-b07e-c8cf64fb34e4-plugins-config-volume\") pod \"612fda7e-6bf2-4179-b07e-c8cf64fb34e4\" (UID: \"612fda7e-6bf2-4179-b07e-c8cf64fb34e4\") "
Sep 04 23:56:15.186676 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.186647    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612fda7e-6bf2-4179-b07e-c8cf64fb34e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "612fda7e-6bf2-4179-b07e-c8cf64fb34e4" (UID: "612fda7e-6bf2-4179-b07e-c8cf64fb34e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:56:15.186987 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.186963    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612fda7e-6bf2-4179-b07e-c8cf64fb34e4-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:56:15.188463 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.188410    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612fda7e-6bf2-4179-b07e-c8cf64fb34e4-kube-api-access-vhxwk" (OuterVolumeSpecName: "kube-api-access-vhxwk") pod "612fda7e-6bf2-4179-b07e-c8cf64fb34e4" (UID: "612fda7e-6bf2-4179-b07e-c8cf64fb34e4"). InnerVolumeSpecName "kube-api-access-vhxwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:56:15.288114 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.288089    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhxwk\" (UniqueName: \"kubernetes.io/projected/612fda7e-6bf2-4179-b07e-c8cf64fb34e4-kube-api-access-vhxwk\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:56:15.395970 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.395886    2557 generic.go:356] "Generic (PLEG): container finished" podID="612fda7e-6bf2-4179-b07e-c8cf64fb34e4" containerID="9d4997285c36dead45935f91ac7d16ed6b84ad653308a46844088413c7f93e02" exitCode=0
Sep 04 23:56:15.395970 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.395928    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"
Sep 04 23:56:15.396101 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.395966    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458" event={"ID":"612fda7e-6bf2-4179-b07e-c8cf64fb34e4","Type":"ContainerDied","Data":"9d4997285c36dead45935f91ac7d16ed6b84ad653308a46844088413c7f93e02"}
Sep 04 23:56:15.396101 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.396024    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458" event={"ID":"612fda7e-6bf2-4179-b07e-c8cf64fb34e4","Type":"ContainerDied","Data":"cb036d03662bb0d7a8a5ded33395e69009ae4e20a4d941761747e5880af8ca0f"}
Sep 04 23:56:15.396101 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.396047    2557 scope.go:117] "RemoveContainer" containerID="9d4997285c36dead45935f91ac7d16ed6b84ad653308a46844088413c7f93e02"
Sep 04 23:56:15.404875 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.404855    2557 scope.go:117] "RemoveContainer" containerID="9d4997285c36dead45935f91ac7d16ed6b84ad653308a46844088413c7f93e02"
Sep 04 23:56:15.405161 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:56:15.405139    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4997285c36dead45935f91ac7d16ed6b84ad653308a46844088413c7f93e02\": container with ID starting with 9d4997285c36dead45935f91ac7d16ed6b84ad653308a46844088413c7f93e02 not found: ID does not exist" containerID="9d4997285c36dead45935f91ac7d16ed6b84ad653308a46844088413c7f93e02"
Sep 04 23:56:15.405224 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.405168    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4997285c36dead45935f91ac7d16ed6b84ad653308a46844088413c7f93e02"} err="failed to get container status \"9d4997285c36dead45935f91ac7d16ed6b84ad653308a46844088413c7f93e02\": rpc error: code = NotFound desc = could not find container \"9d4997285c36dead45935f91ac7d16ed6b84ad653308a46844088413c7f93e02\": container with ID starting with 9d4997285c36dead45935f91ac7d16ed6b84ad653308a46844088413c7f93e02 not found: ID does not exist"
Sep 04 23:56:15.417764 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.417739    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"]
Sep 04 23:56:15.423999 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:15.423979    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ede0daf1-858f7dcdb5-dt458"]
Sep 04 23:56:16.078921 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:16.078892    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="612fda7e-6bf2-4179-b07e-c8cf64fb34e4" path="/var/lib/kubelet/pods/612fda7e-6bf2-4179-b07e-c8cf64fb34e4/volumes"
Sep 04 23:56:16.079289 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:16.079269    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd5666a8-5dca-4af3-b900-8ea30ebd9e5b" path="/var/lib/kubelet/pods/bd5666a8-5dca-4af3-b900-8ea30ebd9e5b/volumes"
Sep 04 23:56:24.719763 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:24.719727    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:56:41.419810 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:41.419769    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:56:57.512949 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:57.512908    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:56:58.099062 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:58.099035    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 04 23:56:58.103366 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:58.103343    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 04 23:56:58.224892 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:58.224863    2557 scope.go:117] "RemoveContainer" containerID="f1d7713249a04898a5674f0acc46c820a9c19a7ab002485ba31c44ed5fe37513"
Sep 04 23:56:58.233693 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:58.233675    2557 scope.go:117] "RemoveContainer" containerID="775583b87e2bd33cc719e2bb43d2e81a844f652b069fdfd838b2407bcdb22799"
Sep 04 23:56:58.241253 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:56:58.241235    2557 scope.go:117] "RemoveContainer" containerID="c2ea612c12e8e5437295551a6d2074fce8f251011250a2013a84b7f4cf294cba"
Sep 04 23:57:05.800223 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:05.800189    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"]
Sep 04 23:57:05.800629 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:05.800428    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc" podUID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" containerName="istio-proxy" containerID="cri-o://3fc126dc7bc5282808dae217db8090551460b98fd17e9bca97ddc7584cee1fda" gracePeriod=30
Sep 04 23:57:09.978591 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:09.978552    2557 patch_prober.go:28] interesting pod/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 04 23:57:09.978974 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:09.978613    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc" podUID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 23:57:10.867975 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:10.867929    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"]
Sep 04 23:57:11.052225 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.052203    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:57:11.175434 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.175353    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-workload-certs\") pod \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") "
Sep 04 23:57:11.175585 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.175434    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67hkl\" (UniqueName: \"kubernetes.io/projected/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-kube-api-access-67hkl\") pod \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") "
Sep 04 23:57:11.175585 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.175501    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-workload-socket\") pod \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") "
Sep 04 23:57:11.175585 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.175527    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-credential-socket\") pod \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") "
Sep 04 23:57:11.175585 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.175563    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-envoy\") pod \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") "
Sep 04 23:57:11.175863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.175589    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istiod-ca-cert\") pod \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") "
Sep 04 23:57:11.175863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.175632    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-podinfo\") pod \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") "
Sep 04 23:57:11.175863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.175637    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4f6fee78-9e89-40fd-a47a-4a97cb7957c9" (UID: "4f6fee78-9e89-40fd-a47a-4a97cb7957c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:57:11.175863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.175683    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-token\") pod \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") "
Sep 04 23:57:11.175863 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.175730    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-data\") pod \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\" (UID: \"4f6fee78-9e89-40fd-a47a-4a97cb7957c9\") "
Sep 04 23:57:11.176153 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.175861    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4f6fee78-9e89-40fd-a47a-4a97cb7957c9" (UID: "4f6fee78-9e89-40fd-a47a-4a97cb7957c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:57:11.176153 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.175880    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4f6fee78-9e89-40fd-a47a-4a97cb7957c9" (UID: "4f6fee78-9e89-40fd-a47a-4a97cb7957c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:57:11.176153 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.176072    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-data" (OuterVolumeSpecName: "istio-data") pod "4f6fee78-9e89-40fd-a47a-4a97cb7957c9" (UID: "4f6fee78-9e89-40fd-a47a-4a97cb7957c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:57:11.176153 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.176091    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:57:11.176153 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.176113    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:57:11.176153 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.176127    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:57:11.176475 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.176326    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4f6fee78-9e89-40fd-a47a-4a97cb7957c9" (UID: "4f6fee78-9e89-40fd-a47a-4a97cb7957c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:57:11.177804 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.177773    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4f6fee78-9e89-40fd-a47a-4a97cb7957c9" (UID: "4f6fee78-9e89-40fd-a47a-4a97cb7957c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:57:11.177962 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.177888    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4f6fee78-9e89-40fd-a47a-4a97cb7957c9" (UID: "4f6fee78-9e89-40fd-a47a-4a97cb7957c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:57:11.178147 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.178124    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-kube-api-access-67hkl" (OuterVolumeSpecName: "kube-api-access-67hkl") pod "4f6fee78-9e89-40fd-a47a-4a97cb7957c9" (UID: "4f6fee78-9e89-40fd-a47a-4a97cb7957c9"). InnerVolumeSpecName "kube-api-access-67hkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:57:11.178208 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.178140    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-token" (OuterVolumeSpecName: "istio-token") pod "4f6fee78-9e89-40fd-a47a-4a97cb7957c9" (UID: "4f6fee78-9e89-40fd-a47a-4a97cb7957c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:57:11.277075 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.277054    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67hkl\" (UniqueName: \"kubernetes.io/projected/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-kube-api-access-67hkl\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:57:11.277075 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.277073    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:57:11.277213 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.277083    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:57:11.277213 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.277091    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:57:11.277213 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.277098    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:57:11.277213 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.277106    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f6fee78-9e89-40fd-a47a-4a97cb7957c9-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:57:11.733685 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.733652    2557 generic.go:356] "Generic (PLEG): container finished" podID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" containerID="3fc126dc7bc5282808dae217db8090551460b98fd17e9bca97ddc7584cee1fda" exitCode=0
Sep 04 23:57:11.733853 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.733707    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"
Sep 04 23:57:11.733853 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.733737    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc" event={"ID":"4f6fee78-9e89-40fd-a47a-4a97cb7957c9","Type":"ContainerDied","Data":"3fc126dc7bc5282808dae217db8090551460b98fd17e9bca97ddc7584cee1fda"}
Sep 04 23:57:11.733853 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.733810    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc" event={"ID":"4f6fee78-9e89-40fd-a47a-4a97cb7957c9","Type":"ContainerDied","Data":"54fdff4284ea9e54cd4c59a867c53f8a42273ed1724a0a72fc6757ec67814883"}
Sep 04 23:57:11.733853 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.733831    2557 scope.go:117] "RemoveContainer" containerID="3fc126dc7bc5282808dae217db8090551460b98fd17e9bca97ddc7584cee1fda"
Sep 04 23:57:11.743110 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.743095    2557 scope.go:117] "RemoveContainer" containerID="440114ab621a3483849021854dfcd6803e74e5c01d151eaa5f96fa9d018fd055"
Sep 04 23:57:11.751342 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.751326    2557 scope.go:117] "RemoveContainer" containerID="3fc126dc7bc5282808dae217db8090551460b98fd17e9bca97ddc7584cee1fda"
Sep 04 23:57:11.751610 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:57:11.751591    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc126dc7bc5282808dae217db8090551460b98fd17e9bca97ddc7584cee1fda\": container with ID starting with 3fc126dc7bc5282808dae217db8090551460b98fd17e9bca97ddc7584cee1fda not found: ID does not exist" containerID="3fc126dc7bc5282808dae217db8090551460b98fd17e9bca97ddc7584cee1fda"
Sep 04 23:57:11.751679 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.751620    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc126dc7bc5282808dae217db8090551460b98fd17e9bca97ddc7584cee1fda"} err="failed to get container status \"3fc126dc7bc5282808dae217db8090551460b98fd17e9bca97ddc7584cee1fda\": rpc error: code = NotFound desc = could not find container \"3fc126dc7bc5282808dae217db8090551460b98fd17e9bca97ddc7584cee1fda\": container with ID starting with 3fc126dc7bc5282808dae217db8090551460b98fd17e9bca97ddc7584cee1fda not found: ID does not exist"
Sep 04 23:57:11.751679 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.751643    2557 scope.go:117] "RemoveContainer" containerID="440114ab621a3483849021854dfcd6803e74e5c01d151eaa5f96fa9d018fd055"
Sep 04 23:57:11.751908 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:57:11.751888    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440114ab621a3483849021854dfcd6803e74e5c01d151eaa5f96fa9d018fd055\": container with ID starting with 440114ab621a3483849021854dfcd6803e74e5c01d151eaa5f96fa9d018fd055 not found: ID does not exist" containerID="440114ab621a3483849021854dfcd6803e74e5c01d151eaa5f96fa9d018fd055"
Sep 04 23:57:11.752090 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.751914    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440114ab621a3483849021854dfcd6803e74e5c01d151eaa5f96fa9d018fd055"} err="failed to get container status \"440114ab621a3483849021854dfcd6803e74e5c01d151eaa5f96fa9d018fd055\": rpc error: code = NotFound desc = could not find container \"440114ab621a3483849021854dfcd6803e74e5c01d151eaa5f96fa9d018fd055\": container with ID starting with 440114ab621a3483849021854dfcd6803e74e5c01d151eaa5f96fa9d018fd055 not found: ID does not exist"
Sep 04 23:57:11.755495 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.755474    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"]
Sep 04 23:57:11.757707 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:11.757687    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ede0daf1-openshift-default-7bc97d95bd-zhbbc"]
Sep 04 23:57:12.079185 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:12.079106    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" path="/var/lib/kubelet/pods/4f6fee78-9e89-40fd-a47a-4a97cb7957c9/volumes"
Sep 04 23:57:13.513370 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:13.513339    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:57:25.887508 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:25.887471    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q" podUID="fa3917c2-789c-410b-a52b-19ef94b2e7b4" containerName="main" containerID="cri-o://5e51968085c85f8ae167f4414b50e84a881cce7ff83ab9f68e038cf098240f66" gracePeriod=45
Sep 04 23:57:26.019019 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.018978    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:57:26.135673 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.135645    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:57:26.300172 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.300144    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-dshm\") pod \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") "
Sep 04 23:57:26.300332 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.300187    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3917c2-789c-410b-a52b-19ef94b2e7b4-tls-certs\") pod \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") "
Sep 04 23:57:26.300332 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.300231    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-model-cache\") pod \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") "
Sep 04 23:57:26.300332 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.300255    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-tmp-dir\") pod \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") "
Sep 04 23:57:26.300332 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.300318    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnkth\" (UniqueName: \"kubernetes.io/projected/fa3917c2-789c-410b-a52b-19ef94b2e7b4-kube-api-access-bnkth\") pod \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") "
Sep 04 23:57:26.300569 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.300341    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-home\") pod \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\" (UID: \"fa3917c2-789c-410b-a52b-19ef94b2e7b4\") "
Sep 04 23:57:26.300569 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.300536    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa3917c2-789c-410b-a52b-19ef94b2e7b4" (UID: "fa3917c2-789c-410b-a52b-19ef94b2e7b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:57:26.300569 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.300552    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa3917c2-789c-410b-a52b-19ef94b2e7b4" (UID: "fa3917c2-789c-410b-a52b-19ef94b2e7b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:57:26.300740 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.300669    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-home" (OuterVolumeSpecName: "home") pod "fa3917c2-789c-410b-a52b-19ef94b2e7b4" (UID: "fa3917c2-789c-410b-a52b-19ef94b2e7b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:57:26.300800 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.300765    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:57:26.300800 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.300778    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:57:26.300800 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.300786    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:57:26.302387 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.302363    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-dshm" (OuterVolumeSpecName: "dshm") pod "fa3917c2-789c-410b-a52b-19ef94b2e7b4" (UID: "fa3917c2-789c-410b-a52b-19ef94b2e7b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:57:26.302511 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.302387    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3917c2-789c-410b-a52b-19ef94b2e7b4-kube-api-access-bnkth" (OuterVolumeSpecName: "kube-api-access-bnkth") pod "fa3917c2-789c-410b-a52b-19ef94b2e7b4" (UID: "fa3917c2-789c-410b-a52b-19ef94b2e7b4"). InnerVolumeSpecName "kube-api-access-bnkth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:57:26.302511 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.302469    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa3917c2-789c-410b-a52b-19ef94b2e7b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa3917c2-789c-410b-a52b-19ef94b2e7b4" (UID: "fa3917c2-789c-410b-a52b-19ef94b2e7b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:57:26.401452 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.401428    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3917c2-789c-410b-a52b-19ef94b2e7b4-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:57:26.401562 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.401453    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnkth\" (UniqueName: \"kubernetes.io/projected/fa3917c2-789c-410b-a52b-19ef94b2e7b4-kube-api-access-bnkth\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:57:26.401562 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.401469    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3917c2-789c-410b-a52b-19ef94b2e7b4-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:57:26.818739 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.818705    2557 generic.go:356] "Generic (PLEG): container finished" podID="fa3917c2-789c-410b-a52b-19ef94b2e7b4" containerID="5e51968085c85f8ae167f4414b50e84a881cce7ff83ab9f68e038cf098240f66" exitCode=0
Sep 04 23:57:26.818926 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.818747    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"
Sep 04 23:57:26.818926 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.818746    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q" event={"ID":"fa3917c2-789c-410b-a52b-19ef94b2e7b4","Type":"ContainerDied","Data":"5e51968085c85f8ae167f4414b50e84a881cce7ff83ab9f68e038cf098240f66"}
Sep 04 23:57:26.818926 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.818800    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q" event={"ID":"fa3917c2-789c-410b-a52b-19ef94b2e7b4","Type":"ContainerDied","Data":"082204accdd35beabcaf397cab2f27ce4e80958e7b6fe3622079253a8f99f79a"}
Sep 04 23:57:26.818926 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.818822    2557 scope.go:117] "RemoveContainer" containerID="5e51968085c85f8ae167f4414b50e84a881cce7ff83ab9f68e038cf098240f66"
Sep 04 23:57:26.831100 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.831076    2557 scope.go:117] "RemoveContainer" containerID="5e51968085c85f8ae167f4414b50e84a881cce7ff83ab9f68e038cf098240f66"
Sep 04 23:57:26.831374 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:57:26.831356    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e51968085c85f8ae167f4414b50e84a881cce7ff83ab9f68e038cf098240f66\": container with ID starting with 5e51968085c85f8ae167f4414b50e84a881cce7ff83ab9f68e038cf098240f66 not found: ID does not exist" containerID="5e51968085c85f8ae167f4414b50e84a881cce7ff83ab9f68e038cf098240f66"
Sep 04 23:57:26.831436 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.831382    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e51968085c85f8ae167f4414b50e84a881cce7ff83ab9f68e038cf098240f66"} err="failed to get container status \"5e51968085c85f8ae167f4414b50e84a881cce7ff83ab9f68e038cf098240f66\": rpc error: code = NotFound desc = could not find container \"5e51968085c85f8ae167f4414b50e84a881cce7ff83ab9f68e038cf098240f66\": container with ID starting with 5e51968085c85f8ae167f4414b50e84a881cce7ff83ab9f68e038cf098240f66 not found: ID does not exist"
Sep 04 23:57:26.844805 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.844781    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"]
Sep 04 23:57:26.847394 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:26.847373    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5245c1/auth-test-model-5245c1-kserve-d7d496fd7-bpc5q"]
Sep 04 23:57:28.085556 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:28.085514    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa3917c2-789c-410b-a52b-19ef94b2e7b4" path="/var/lib/kubelet/pods/fa3917c2-789c-410b-a52b-19ef94b2e7b4/volumes"
Sep 04 23:57:39.311699 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:39.311665    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:57:55.524121 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524024    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"]
Sep 04 23:57:55.524690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524421    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="612fda7e-6bf2-4179-b07e-c8cf64fb34e4" containerName="payload-processing"
Sep 04 23:57:55.524690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524438    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="612fda7e-6bf2-4179-b07e-c8cf64fb34e4" containerName="payload-processing"
Sep 04 23:57:55.524690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524459    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" containerName="istio-proxy"
Sep 04 23:57:55.524690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524464    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" containerName="istio-proxy"
Sep 04 23:57:55.524690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524473    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" containerName="istio-proxy"
Sep 04 23:57:55.524690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524478    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" containerName="istio-proxy"
Sep 04 23:57:55.524690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524485    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa3917c2-789c-410b-a52b-19ef94b2e7b4" containerName="main"
Sep 04 23:57:55.524690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524491    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa3917c2-789c-410b-a52b-19ef94b2e7b4" containerName="main"
Sep 04 23:57:55.524690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524502    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd5666a8-5dca-4af3-b900-8ea30ebd9e5b" containerName="revoke-keys"
Sep 04 23:57:55.524690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524510    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd5666a8-5dca-4af3-b900-8ea30ebd9e5b" containerName="revoke-keys"
Sep 04 23:57:55.524690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524568    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" containerName="istio-proxy"
Sep 04 23:57:55.524690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524576    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f6fee78-9e89-40fd-a47a-4a97cb7957c9" containerName="istio-proxy"
Sep 04 23:57:55.524690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524586    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa3917c2-789c-410b-a52b-19ef94b2e7b4" containerName="main"
Sep 04 23:57:55.524690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524594    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="612fda7e-6bf2-4179-b07e-c8cf64fb34e4" containerName="payload-processing"
Sep 04 23:57:55.524690 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.524602    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd5666a8-5dca-4af3-b900-8ea30ebd9e5b" containerName="revoke-keys"
Sep 04 23:57:55.528010 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.527987    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.530896 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.530872    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-88828d\"/\"kube-root-ca.crt\""
Sep 04 23:57:55.531050 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.530873    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-88828d\"/\"default-dockercfg-w5fgz\""
Sep 04 23:57:55.532303 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.532280    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-88828d\"/\"openshift-service-ca.crt\""
Sep 04 23:57:55.532472 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.532284    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-88828d\"/\"sub-test-model-88828d-kserve-self-signed-certs\""
Sep 04 23:57:55.536576 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.536552    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"]
Sep 04 23:57:55.653351 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.653318    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-dshm\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.653522 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.653385    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm4vt\" (UniqueName: \"kubernetes.io/projected/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-kube-api-access-rm4vt\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.653522 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.653458    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-tmp-dir\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.653522 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.653506    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-model-cache\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.653639 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.653548    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-home\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.653639 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.653596    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-tls-certs\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.754426 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.754398    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-tls-certs\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.754554 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.754433    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-dshm\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.754554 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.754469    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm4vt\" (UniqueName: \"kubernetes.io/projected/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-kube-api-access-rm4vt\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.754554 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.754500    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-tmp-dir\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.754554 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.754528    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-model-cache\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.754799 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.754555    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-home\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.754888 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.754860    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-tmp-dir\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.755036 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.754932    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-model-cache\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.755036 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.754967    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-home\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.756724 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.756707    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-dshm\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.757021 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.757004    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-tls-certs\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.762072 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.762055    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm4vt\" (UniqueName: \"kubernetes.io/projected/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-kube-api-access-rm4vt\") pod \"sub-test-model-88828d-kserve-85669cf47f-nxx72\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") " pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:55.839253 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:55.839160    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:57:56.110837 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:56.110803    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"]
Sep 04 23:57:56.112185 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:57:56.112160    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fc6aade_57c0_45f5_b4cf_e5c68c10ae50.slice/crio-4a131b21165a444628ceae8fa07356edcf15f46526641e2018110994f6e65bb7 WatchSource:0}: Error finding container 4a131b21165a444628ceae8fa07356edcf15f46526641e2018110994f6e65bb7: Status 404 returned error can't find the container with id 4a131b21165a444628ceae8fa07356edcf15f46526641e2018110994f6e65bb7
Sep 04 23:57:56.994733 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:56.994703    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72" event={"ID":"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50","Type":"ContainerStarted","Data":"2e8233167fc5aa4beecc53cf8d4b80e1ce28cd78de43e809e8b8b5a7747f22fe"}
Sep 04 23:57:56.995096 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:56.994742    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72" event={"ID":"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50","Type":"ContainerStarted","Data":"4a131b21165a444628ceae8fa07356edcf15f46526641e2018110994f6e65bb7"}
Sep 04 23:57:57.012957 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:57.012905    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72" podStartSLOduration=2.012888241 podStartE2EDuration="2.012888241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:57:57.009363253 +0000 UTC m=+1259.427181583" watchObservedRunningTime="2026-09-04 23:57:57.012888241 +0000 UTC m=+1259.430706561"
Sep 04 23:57:59.027683 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:59.027647    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:57:59.939658 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:59.939626    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"]
Sep 04 23:57:59.944500 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:59.944478    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:57:59.947167 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:59.947150    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-87b0d1-openshift-default-dockercfg-zp968\""
Sep 04 23:57:59.953501 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:59.953477    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"]
Sep 04 23:57:59.990978 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:59.990926    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-credential-socket\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:57:59.991089 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:59.990987    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-data\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:57:59.991089 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:59.991033    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-podinfo\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:57:59.991089 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:59.991057    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-workload-certs\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:57:59.991089 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:59.991077    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-envoy\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:57:59.991231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:59.991121    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-workload-socket\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:57:59.991231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:59.991145    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xt5t\" (UniqueName: \"kubernetes.io/projected/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-kube-api-access-8xt5t\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:57:59.991231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:59.991200    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istiod-ca-cert\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:57:59.991231 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:57:59.991222    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-token\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.092986 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.092823    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-workload-socket\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.092986 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.092876    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xt5t\" (UniqueName: \"kubernetes.io/projected/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-kube-api-access-8xt5t\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.092986 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.092932    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istiod-ca-cert\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.092986 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.092989    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-token\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.093638 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.093077    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-credential-socket\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.093638 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.093102    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-data\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.093638 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.093151    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-podinfo\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.093638 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.093189    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-workload-certs\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.093854 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.093639    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-workload-certs\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.096914 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.093905    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-workload-socket\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.096914 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.094832    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istiod-ca-cert\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.096914 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.095345    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-credential-socket\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.096914 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.095592    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-data\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.096914 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.096006    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-envoy\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.100130 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.099216    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-podinfo\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.107140 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.107092    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-envoy\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.109652 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.109624    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-token\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.110534 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.110460    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xt5t\" (UniqueName: \"kubernetes.io/projected/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-kube-api-access-8xt5t\") pod \"e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") " pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.256450 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.256405    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:00.550114 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.550088    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"]
Sep 04 23:58:00.550832 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:58:00.550809    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed6db1e_25f9_45b6_a94d_ff6d4c24362d.slice/crio-256db219d7dc997ff52189c61a9c8280d7e19a0dce47c1e81112e966a0a398c3 WatchSource:0}: Error finding container 256db219d7dc997ff52189c61a9c8280d7e19a0dce47c1e81112e966a0a398c3: Status 404 returned error can't find the container with id 256db219d7dc997ff52189c61a9c8280d7e19a0dce47c1e81112e966a0a398c3
Sep 04 23:58:00.553558 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.553517    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:58:00.553643 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.553586    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:58:00.553643 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:00.553614    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:58:01.020951 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:01.020917    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" event={"ID":"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d","Type":"ContainerStarted","Data":"670ad988b1902e651b7a3bbd14bc4ee839d55baedd4485490bc2c0b39f873965"}
Sep 04 23:58:01.021114 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:01.020990    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" event={"ID":"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d","Type":"ContainerStarted","Data":"256db219d7dc997ff52189c61a9c8280d7e19a0dce47c1e81112e966a0a398c3"}
Sep 04 23:58:01.041889 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:01.041837    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" podStartSLOduration=2.04182353 podStartE2EDuration="2.04182353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:58:01.040131534 +0000 UTC m=+1263.457949852" watchObservedRunningTime="2026-09-04 23:58:01.04182353 +0000 UTC m=+1263.459641849"
Sep 04 23:58:01.257203 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:01.257172    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:01.262749 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:01.262728    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:02.026307 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:02.026271    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:02.027082 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:02.027062    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:05.840167 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:05.840138    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:58:05.840526 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:05.840179    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:58:05.852899 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:05.852879    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:58:06.059734 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:06.059711    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:58:18.068085 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:18.068045    2557 patch_prober.go:28] interesting pod/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 04 23:58:18.068551 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:18.068135    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 04 23:58:19.142227 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:19.142193    2557 generic.go:356] "Generic (PLEG): container finished" podID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerID="670ad988b1902e651b7a3bbd14bc4ee839d55baedd4485490bc2c0b39f873965" exitCode=0
Sep 04 23:58:19.142651 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:19.142267    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" event={"ID":"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d","Type":"ContainerDied","Data":"670ad988b1902e651b7a3bbd14bc4ee839d55baedd4485490bc2c0b39f873965"}
Sep 04 23:58:19.142697 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:19.142656    2557 scope.go:117] "RemoveContainer" containerID="670ad988b1902e651b7a3bbd14bc4ee839d55baedd4485490bc2c0b39f873965"
Sep 04 23:58:19.143716 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:19.143685    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:58:19.143777 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:19.143753    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:58:19.143820 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:19.143786    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:58:20.149275 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:20.149239    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" event={"ID":"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d","Type":"ContainerStarted","Data":"bcd256ee11b804831c67faed3ede934e51e6b32d74d23813e0b1b90577618d47"}
Sep 04 23:58:20.256826 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:20.256795    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:20.257111 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:20.257091    2557 patch_prober.go:28] interesting pod/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 04 23:58:20.257162 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:20.257132    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 04 23:58:20.921610 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:20.921565    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:58:21.257667 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:21.257636    2557 patch_prober.go:28] interesting pod/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 04 23:58:21.258085 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:21.257688    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 04 23:58:22.257657 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:22.257624    2557 patch_prober.go:28] interesting pod/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 04 23:58:22.257843 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:22.257679    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 04 23:58:23.256785 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:23.256754    2557 patch_prober.go:28] interesting pod/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 04 23:58:23.256980 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:23.256805    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 04 23:58:24.257648 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:24.257616    2557 patch_prober.go:28] interesting pod/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 04 23:58:24.258008 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:24.257669    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 04 23:58:25.257203 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:25.257173    2557 patch_prober.go:28] interesting pod/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 04 23:58:25.257393 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:25.257222    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 04 23:58:26.260680 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:26.260643    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:26.261259 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:26.261046    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:26.261643 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:26.261624    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:58:33.017332 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.017300    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"]
Sep 04 23:58:33.021056 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.021039    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"
Sep 04 23:58:33.023665 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.023645    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8cff3b\""
Sep 04 23:58:33.023797 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.023685    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8cff3b-dockercfg-8522n\""
Sep 04 23:58:33.034832 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.034804    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"]
Sep 04 23:58:33.050441 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.050412    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"]
Sep 04 23:58:33.054476 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.054454    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"
Sep 04 23:58:33.063155 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.063131    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"]
Sep 04 23:58:33.099390 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.099362    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkkww\" (UniqueName: \"kubernetes.io/projected/22cc7e55-5fbd-4ddf-89a7-0669d160ffa7-kube-api-access-gkkww\") pod \"payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs\" (UID: \"22cc7e55-5fbd-4ddf-89a7-0669d160ffa7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"
Sep 04 23:58:33.099529 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.099509    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22cc7e55-5fbd-4ddf-89a7-0669d160ffa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs\" (UID: \"22cc7e55-5fbd-4ddf-89a7-0669d160ffa7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"
Sep 04 23:58:33.099724 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.099700    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86ntz\" (UniqueName: \"kubernetes.io/projected/d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af-kube-api-access-86ntz\") pod \"payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx\" (UID: \"d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"
Sep 04 23:58:33.099838 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.099820    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx\" (UID: \"d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"
Sep 04 23:58:33.200637 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.200596    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22cc7e55-5fbd-4ddf-89a7-0669d160ffa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs\" (UID: \"22cc7e55-5fbd-4ddf-89a7-0669d160ffa7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"
Sep 04 23:58:33.200637 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.200645    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86ntz\" (UniqueName: \"kubernetes.io/projected/d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af-kube-api-access-86ntz\") pod \"payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx\" (UID: \"d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"
Sep 04 23:58:33.201116 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.200703    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx\" (UID: \"d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"
Sep 04 23:58:33.201116 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.200743    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkkww\" (UniqueName: \"kubernetes.io/projected/22cc7e55-5fbd-4ddf-89a7-0669d160ffa7-kube-api-access-gkkww\") pod \"payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs\" (UID: \"22cc7e55-5fbd-4ddf-89a7-0669d160ffa7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"
Sep 04 23:58:33.201472 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.201447    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx\" (UID: \"d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"
Sep 04 23:58:33.201554 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.201479    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22cc7e55-5fbd-4ddf-89a7-0669d160ffa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs\" (UID: \"22cc7e55-5fbd-4ddf-89a7-0669d160ffa7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"
Sep 04 23:58:33.210716 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.210695    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkkww\" (UniqueName: \"kubernetes.io/projected/22cc7e55-5fbd-4ddf-89a7-0669d160ffa7-kube-api-access-gkkww\") pod \"payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs\" (UID: \"22cc7e55-5fbd-4ddf-89a7-0669d160ffa7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"
Sep 04 23:58:33.211747 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.211727    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86ntz\" (UniqueName: \"kubernetes.io/projected/d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af-kube-api-access-86ntz\") pod \"payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx\" (UID: \"d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"
Sep 04 23:58:33.331487 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.331390    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"
Sep 04 23:58:33.366942 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.366917    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"
Sep 04 23:58:33.614197 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.614169    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"]
Sep 04 23:58:33.614877 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:58:33.614853    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cc7e55_5fbd_4ddf_89a7_0669d160ffa7.slice/crio-bcbbbb36d62f63fe4124ffcd354b05b0d48cf20abc7ba0a8df964cd108ca4ed5 WatchSource:0}: Error finding container bcbbbb36d62f63fe4124ffcd354b05b0d48cf20abc7ba0a8df964cd108ca4ed5: Status 404 returned error can't find the container with id bcbbbb36d62f63fe4124ffcd354b05b0d48cf20abc7ba0a8df964cd108ca4ed5
Sep 04 23:58:33.655618 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:33.655594    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"]
Sep 04 23:58:33.656218 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:58:33.656195    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e9c16d_1e7b_4b6a_a836_28a3bd3d33af.slice/crio-2371f8a1db785dbb1bb31f8b61c9c1f1831098bcbf65151096df7ee6ff9643e5 WatchSource:0}: Error finding container 2371f8a1db785dbb1bb31f8b61c9c1f1831098bcbf65151096df7ee6ff9643e5: Status 404 returned error can't find the container with id 2371f8a1db785dbb1bb31f8b61c9c1f1831098bcbf65151096df7ee6ff9643e5
Sep 04 23:58:34.230615 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:34.230551    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx" event={"ID":"d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af","Type":"ContainerStarted","Data":"3fd269d8874b2d9e94eea7328b2e57d8d0bcfccd91868a0e76bc7b8ed43e5570"}
Sep 04 23:58:34.230615 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:34.230610    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"
Sep 04 23:58:34.231132 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:34.230635    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx" event={"ID":"d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af","Type":"ContainerStarted","Data":"2371f8a1db785dbb1bb31f8b61c9c1f1831098bcbf65151096df7ee6ff9643e5"}
Sep 04 23:58:34.232364 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:34.232339    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs" event={"ID":"22cc7e55-5fbd-4ddf-89a7-0669d160ffa7","Type":"ContainerStarted","Data":"6e4b9f382a588160471d52dbbc146de30f1d21a1796861579bfddc5b272d68a4"}
Sep 04 23:58:34.232496 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:34.232383    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs" event={"ID":"22cc7e55-5fbd-4ddf-89a7-0669d160ffa7","Type":"ContainerStarted","Data":"bcbbbb36d62f63fe4124ffcd354b05b0d48cf20abc7ba0a8df964cd108ca4ed5"}
Sep 04 23:58:34.232496 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:34.232419    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"
Sep 04 23:58:34.248635 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:34.248583    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx" podStartSLOduration=1.248565612 podStartE2EDuration="1.248565612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:58:34.245744838 +0000 UTC m=+1296.663563156" watchObservedRunningTime="2026-09-04 23:58:34.248565612 +0000 UTC m=+1296.666383930"
Sep 04 23:58:34.262643 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:34.262601    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs" podStartSLOduration=2.262588183 podStartE2EDuration="2.262588183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:58:34.260042578 +0000 UTC m=+1296.677860898" watchObservedRunningTime="2026-09-04 23:58:34.262588183 +0000 UTC m=+1296.680406500"
Sep 04 23:58:41.008907 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:41.008868    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"]
Sep 04 23:58:42.838215 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:42.838177    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:58:45.240727 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:45.240690    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"
Sep 04 23:58:45.241142 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:45.241041    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"
Sep 04 23:58:56.028711 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.028639    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72" podUID="0fc6aade-57c0-45f5-b4cf-e5c68c10ae50" containerName="main" containerID="cri-o://2e8233167fc5aa4beecc53cf8d4b80e1ce28cd78de43e809e8b8b5a7747f22fe" gracePeriod=45
Sep 04 23:58:56.049541 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.049509    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72" podUID="0fc6aade-57c0-45f5-b4cf-e5c68c10ae50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 04 23:58:56.279249 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.279196    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:58:56.361855 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.361825    2557 generic.go:356] "Generic (PLEG): container finished" podID="0fc6aade-57c0-45f5-b4cf-e5c68c10ae50" containerID="2e8233167fc5aa4beecc53cf8d4b80e1ce28cd78de43e809e8b8b5a7747f22fe" exitCode=0
Sep 04 23:58:56.362016 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.361865    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"
Sep 04 23:58:56.362016 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.361870    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72" event={"ID":"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50","Type":"ContainerDied","Data":"2e8233167fc5aa4beecc53cf8d4b80e1ce28cd78de43e809e8b8b5a7747f22fe"}
Sep 04 23:58:56.362016 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.361896    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72" event={"ID":"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50","Type":"ContainerDied","Data":"4a131b21165a444628ceae8fa07356edcf15f46526641e2018110994f6e65bb7"}
Sep 04 23:58:56.362016 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.361912    2557 scope.go:117] "RemoveContainer" containerID="2e8233167fc5aa4beecc53cf8d4b80e1ce28cd78de43e809e8b8b5a7747f22fe"
Sep 04 23:58:56.370702 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.370684    2557 scope.go:117] "RemoveContainer" containerID="2e8233167fc5aa4beecc53cf8d4b80e1ce28cd78de43e809e8b8b5a7747f22fe"
Sep 04 23:58:56.371008 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:58:56.370987    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8233167fc5aa4beecc53cf8d4b80e1ce28cd78de43e809e8b8b5a7747f22fe\": container with ID starting with 2e8233167fc5aa4beecc53cf8d4b80e1ce28cd78de43e809e8b8b5a7747f22fe not found: ID does not exist" containerID="2e8233167fc5aa4beecc53cf8d4b80e1ce28cd78de43e809e8b8b5a7747f22fe"
Sep 04 23:58:56.371070 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.371017    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8233167fc5aa4beecc53cf8d4b80e1ce28cd78de43e809e8b8b5a7747f22fe"} err="failed to get container status \"2e8233167fc5aa4beecc53cf8d4b80e1ce28cd78de43e809e8b8b5a7747f22fe\": rpc error: code = NotFound desc = could not find container \"2e8233167fc5aa4beecc53cf8d4b80e1ce28cd78de43e809e8b8b5a7747f22fe\": container with ID starting with 2e8233167fc5aa4beecc53cf8d4b80e1ce28cd78de43e809e8b8b5a7747f22fe not found: ID does not exist"
Sep 04 23:58:56.404384 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.404356    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-dshm\") pod \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") "
Sep 04 23:58:56.404498 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.404430    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-tls-certs\") pod \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") "
Sep 04 23:58:56.404563 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.404499    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-home\") pod \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") "
Sep 04 23:58:56.404563 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.404547    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm4vt\" (UniqueName: \"kubernetes.io/projected/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-kube-api-access-rm4vt\") pod \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") "
Sep 04 23:58:56.404671 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.404603    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-model-cache\") pod \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") "
Sep 04 23:58:56.404671 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.404660    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-tmp-dir\") pod \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\" (UID: \"0fc6aade-57c0-45f5-b4cf-e5c68c10ae50\") "
Sep 04 23:58:56.404774 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.404730    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-home" (OuterVolumeSpecName: "home") pod "0fc6aade-57c0-45f5-b4cf-e5c68c10ae50" (UID: "0fc6aade-57c0-45f5-b4cf-e5c68c10ae50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:58:56.404958 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.404917    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0fc6aade-57c0-45f5-b4cf-e5c68c10ae50" (UID: "0fc6aade-57c0-45f5-b4cf-e5c68c10ae50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:58:56.405067 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.404951    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-model-cache" (OuterVolumeSpecName: "model-cache") pod "0fc6aade-57c0-45f5-b4cf-e5c68c10ae50" (UID: "0fc6aade-57c0-45f5-b4cf-e5c68c10ae50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:58:56.405067 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.405020    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:58:56.406528 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.406501    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fc6aade-57c0-45f5-b4cf-e5c68c10ae50" (UID: "0fc6aade-57c0-45f5-b4cf-e5c68c10ae50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:58:56.406611 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.406553    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-dshm" (OuterVolumeSpecName: "dshm") pod "0fc6aade-57c0-45f5-b4cf-e5c68c10ae50" (UID: "0fc6aade-57c0-45f5-b4cf-e5c68c10ae50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:58:56.406611 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.406570    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-kube-api-access-rm4vt" (OuterVolumeSpecName: "kube-api-access-rm4vt") pod "0fc6aade-57c0-45f5-b4cf-e5c68c10ae50" (UID: "0fc6aade-57c0-45f5-b4cf-e5c68c10ae50"). InnerVolumeSpecName "kube-api-access-rm4vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:58:56.505862 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.505836    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:58:56.505862 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.505857    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:58:56.506022 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.505869    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:58:56.506022 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.505877    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:58:56.506022 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.505885    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm4vt\" (UniqueName: \"kubernetes.io/projected/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50-kube-api-access-rm4vt\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:58:56.683720 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.683692    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"]
Sep 04 23:58:56.690646 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:56.690619    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-88828d/sub-test-model-88828d-kserve-85669cf47f-nxx72"]
Sep 04 23:58:58.079080 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:58.079042    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fc6aade-57c0-45f5-b4cf-e5c68c10ae50" path="/var/lib/kubelet/pods/0fc6aade-57c0-45f5-b4cf-e5c68c10ae50/volumes"
Sep 04 23:58:58.291916 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:58.291885    2557 scope.go:117] "RemoveContainer" containerID="6ab110b8cb7cbf2d4c2f0e7cf2f7e7d1f553e0ba32357065143e103aeac5dc4c"
Sep 04 23:58:58.300277 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:58:58.300261    2557 scope.go:117] "RemoveContainer" containerID="cd0c61371a5a431ec319f864d1583f3cfd26ffeef202a4b1642a5eae4b4742ab"
Sep 04 23:59:04.911276 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:04.911244    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:59:10.224450 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.224419    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"]
Sep 04 23:59:10.224889 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.224873    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc6aade-57c0-45f5-b4cf-e5c68c10ae50" containerName="main"
Sep 04 23:59:10.224965 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.224892    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc6aade-57c0-45f5-b4cf-e5c68c10ae50" containerName="main"
Sep 04 23:59:10.225017 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.224995    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fc6aade-57c0-45f5-b4cf-e5c68c10ae50" containerName="main"
Sep 04 23:59:10.230084 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.230064    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.233079 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.233045    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-88828d\"/\"openshift-service-ca.crt\""
Sep 04 23:59:10.234249 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.234228    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-88828d\"/\"default-dockercfg-w5fgz\""
Sep 04 23:59:10.234508 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.234488    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-88828d\"/\"kube-root-ca.crt\""
Sep 04 23:59:10.234628 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.234606    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-88828d\"/\"rate-test-model-88828d-kserve-self-signed-certs\""
Sep 04 23:59:10.243834 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.243808    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"]
Sep 04 23:59:10.324846 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.324821    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-tmp-dir\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.324996 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.324855    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-dshm\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.324996 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.324907    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-home\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.324996 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.324923    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-tls-certs\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.325128 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.325002    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjjj9\" (UniqueName: \"kubernetes.io/projected/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-kube-api-access-wjjj9\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.325128 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.325090    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-model-cache\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.426151 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.426118    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-home\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.426151 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.426154    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-tls-certs\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.426368 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.426190    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjjj9\" (UniqueName: \"kubernetes.io/projected/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-kube-api-access-wjjj9\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.426368 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.426260    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-model-cache\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.426368 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.426306    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-tmp-dir\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.426368 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.426337    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-dshm\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.426668 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.426638    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-home\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.426764 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.426675    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-model-cache\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.426821 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.426799    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-tmp-dir\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.428596 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.428572    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-dshm\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.428779 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.428762    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-tls-certs\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.433827 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.433806    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjjj9\" (UniqueName: \"kubernetes.io/projected/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-kube-api-access-wjjj9\") pod \"rate-test-model-88828d-kserve-67bbcf84f9-c2jk2\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") " pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.555331 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.555241    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:10.827335 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:10.827310    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"]
Sep 04 23:59:10.828011 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:59:10.827975    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f12385e_73c9_4f0d_a5e7_82e3ce8cf675.slice/crio-ed70c1ec1938d2c4047108959c3b54d794597777db0c091718a1a20695dd5285 WatchSource:0}: Error finding container ed70c1ec1938d2c4047108959c3b54d794597777db0c091718a1a20695dd5285: Status 404 returned error can't find the container with id ed70c1ec1938d2c4047108959c3b54d794597777db0c091718a1a20695dd5285
Sep 04 23:59:11.455670 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:11.455638    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2" event={"ID":"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675","Type":"ContainerStarted","Data":"3cc20d628238019164b775eba275c505da4a7c471f795d53ff1a9d1eef56cc19"}
Sep 04 23:59:11.456054 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:11.455680    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2" event={"ID":"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675","Type":"ContainerStarted","Data":"ed70c1ec1938d2c4047108959c3b54d794597777db0c091718a1a20695dd5285"}
Sep 04 23:59:11.472084 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:11.472041    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2" podStartSLOduration=1.4720272030000001 podStartE2EDuration="1.472027203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:59:11.470451222 +0000 UTC m=+1333.888269540" watchObservedRunningTime="2026-09-04 23:59:11.472027203 +0000 UTC m=+1333.889845515"
Sep 04 23:59:18.722947 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:18.722902    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:59:20.555919 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:20.555839    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:20.555919 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:20.555886    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:20.568421 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:20.568394    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:21.216225 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:21.216191    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"]
Sep 04 23:59:21.216484 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:21.216456    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerName="istio-proxy" containerID="cri-o://bcd256ee11b804831c67faed3ede934e51e6b32d74d23813e0b1b90577618d47" gracePeriod=30
Sep 04 23:59:21.522924 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:21.522849    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 04 23:59:26.262604 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.262574    2557 patch_prober.go:28] interesting pod/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 04 23:59:26.262890 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.262629    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 04 23:59:26.471468 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.471442    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:59:26.540859 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.540799    2557 generic.go:356] "Generic (PLEG): container finished" podID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerID="bcd256ee11b804831c67faed3ede934e51e6b32d74d23813e0b1b90577618d47" exitCode=0
Sep 04 23:59:26.540859 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.540847    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"
Sep 04 23:59:26.541043 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.540888    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" event={"ID":"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d","Type":"ContainerDied","Data":"bcd256ee11b804831c67faed3ede934e51e6b32d74d23813e0b1b90577618d47"}
Sep 04 23:59:26.541043 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.540956    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr" event={"ID":"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d","Type":"ContainerDied","Data":"256db219d7dc997ff52189c61a9c8280d7e19a0dce47c1e81112e966a0a398c3"}
Sep 04 23:59:26.541043 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.540991    2557 scope.go:117] "RemoveContainer" containerID="bcd256ee11b804831c67faed3ede934e51e6b32d74d23813e0b1b90577618d47"
Sep 04 23:59:26.549820 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.549803    2557 scope.go:117] "RemoveContainer" containerID="670ad988b1902e651b7a3bbd14bc4ee839d55baedd4485490bc2c0b39f873965"
Sep 04 23:59:26.558058 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.558034    2557 scope.go:117] "RemoveContainer" containerID="bcd256ee11b804831c67faed3ede934e51e6b32d74d23813e0b1b90577618d47"
Sep 04 23:59:26.558338 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:59:26.558316    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd256ee11b804831c67faed3ede934e51e6b32d74d23813e0b1b90577618d47\": container with ID starting with bcd256ee11b804831c67faed3ede934e51e6b32d74d23813e0b1b90577618d47 not found: ID does not exist" containerID="bcd256ee11b804831c67faed3ede934e51e6b32d74d23813e0b1b90577618d47"
Sep 04 23:59:26.558441 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.558342    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd256ee11b804831c67faed3ede934e51e6b32d74d23813e0b1b90577618d47"} err="failed to get container status \"bcd256ee11b804831c67faed3ede934e51e6b32d74d23813e0b1b90577618d47\": rpc error: code = NotFound desc = could not find container \"bcd256ee11b804831c67faed3ede934e51e6b32d74d23813e0b1b90577618d47\": container with ID starting with bcd256ee11b804831c67faed3ede934e51e6b32d74d23813e0b1b90577618d47 not found: ID does not exist"
Sep 04 23:59:26.558441 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.558360    2557 scope.go:117] "RemoveContainer" containerID="670ad988b1902e651b7a3bbd14bc4ee839d55baedd4485490bc2c0b39f873965"
Sep 04 23:59:26.558600 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:59:26.558581    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670ad988b1902e651b7a3bbd14bc4ee839d55baedd4485490bc2c0b39f873965\": container with ID starting with 670ad988b1902e651b7a3bbd14bc4ee839d55baedd4485490bc2c0b39f873965 not found: ID does not exist" containerID="670ad988b1902e651b7a3bbd14bc4ee839d55baedd4485490bc2c0b39f873965"
Sep 04 23:59:26.558643 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.558609    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670ad988b1902e651b7a3bbd14bc4ee839d55baedd4485490bc2c0b39f873965"} err="failed to get container status \"670ad988b1902e651b7a3bbd14bc4ee839d55baedd4485490bc2c0b39f873965\": rpc error: code = NotFound desc = could not find container \"670ad988b1902e651b7a3bbd14bc4ee839d55baedd4485490bc2c0b39f873965\": container with ID starting with 670ad988b1902e651b7a3bbd14bc4ee839d55baedd4485490bc2c0b39f873965 not found: ID does not exist"
Sep 04 23:59:26.558643 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.558635    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-podinfo\") pod \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") "
Sep 04 23:59:26.558722 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.558694    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-credential-socket\") pod \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") "
Sep 04 23:59:26.558773 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.558759    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-token\") pod \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") "
Sep 04 23:59:26.558820 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.558792    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xt5t\" (UniqueName: \"kubernetes.io/projected/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-kube-api-access-8xt5t\") pod \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") "
Sep 04 23:59:26.558857 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.558826    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-data\") pod \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") "
Sep 04 23:59:26.558900 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.558880    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-workload-certs\") pod \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") "
Sep 04 23:59:26.558980 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.558905    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-envoy\") pod \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") "
Sep 04 23:59:26.559049 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.558985    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-workload-socket\") pod \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") "
Sep 04 23:59:26.559049 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.558991    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" (UID: "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:59:26.559049 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.559012    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istiod-ca-cert\") pod \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\" (UID: \"3ed6db1e-25f9-45b6-a94d-ff6d4c24362d\") "
Sep 04 23:59:26.559218 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.559148    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" (UID: "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:59:26.559395 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.559367    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" (UID: "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:59:26.559478 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.559382    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:26.559478 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.559426    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:26.559478 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.559313    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" (UID: "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:59:26.559478 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.559453    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" (UID: "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:59:26.560840 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.560811    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" (UID: "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:59:26.561100 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.561073    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" (UID: "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:59:26.561304 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.561278    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-kube-api-access-8xt5t" (OuterVolumeSpecName: "kube-api-access-8xt5t") pod "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" (UID: "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d"). InnerVolumeSpecName "kube-api-access-8xt5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:59:26.561405 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.561344    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" (UID: "3ed6db1e-25f9-45b6-a94d-ff6d4c24362d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:59:26.660129 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.660105    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:26.660129 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.660126    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xt5t\" (UniqueName: \"kubernetes.io/projected/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-kube-api-access-8xt5t\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:26.660257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.660136    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:26.660257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.660146    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:26.660257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.660154    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:26.660257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.660162    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:26.660257 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.660169    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:26.862341 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.862305    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"]
Sep 04 23:59:26.866540 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:26.866507    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-87b0d1-openshift-default-6b785cf79b-khrmr"]
Sep 04 23:59:28.079913 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:28.079877    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" path="/var/lib/kubelet/pods/3ed6db1e-25f9-45b6-a94d-ff6d4c24362d/volumes"
Sep 04 23:59:37.261387 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.261354    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"]
Sep 04 23:59:37.262016 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.261641    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx" podUID="d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af" containerName="payload-processing" containerID="cri-o://3fd269d8874b2d9e94eea7328b2e57d8d0bcfccd91868a0e76bc7b8ed43e5570" gracePeriod=30
Sep 04 23:59:37.270363 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.270332    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"]
Sep 04 23:59:37.270567 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.270537    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs" podUID="22cc7e55-5fbd-4ddf-89a7-0669d160ffa7" containerName="payload-pre-processing" containerID="cri-o://6e4b9f382a588160471d52dbbc146de30f1d21a1796861579bfddc5b272d68a4" gracePeriod=30
Sep 04 23:59:37.536678 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.536652    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"
Sep 04 23:59:37.563903 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.563879    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"
Sep 04 23:59:37.617777 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.617741    2557 generic.go:356] "Generic (PLEG): container finished" podID="d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af" containerID="3fd269d8874b2d9e94eea7328b2e57d8d0bcfccd91868a0e76bc7b8ed43e5570" exitCode=0
Sep 04 23:59:37.617976 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.617785    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"
Sep 04 23:59:37.617976 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.617829    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx" event={"ID":"d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af","Type":"ContainerDied","Data":"3fd269d8874b2d9e94eea7328b2e57d8d0bcfccd91868a0e76bc7b8ed43e5570"}
Sep 04 23:59:37.617976 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.617880    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx" event={"ID":"d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af","Type":"ContainerDied","Data":"2371f8a1db785dbb1bb31f8b61c9c1f1831098bcbf65151096df7ee6ff9643e5"}
Sep 04 23:59:37.617976 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.617907    2557 scope.go:117] "RemoveContainer" containerID="3fd269d8874b2d9e94eea7328b2e57d8d0bcfccd91868a0e76bc7b8ed43e5570"
Sep 04 23:59:37.619184 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.619162    2557 generic.go:356] "Generic (PLEG): container finished" podID="22cc7e55-5fbd-4ddf-89a7-0669d160ffa7" containerID="6e4b9f382a588160471d52dbbc146de30f1d21a1796861579bfddc5b272d68a4" exitCode=0
Sep 04 23:59:37.619300 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.619202    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"
Sep 04 23:59:37.619300 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.619242    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs" event={"ID":"22cc7e55-5fbd-4ddf-89a7-0669d160ffa7","Type":"ContainerDied","Data":"6e4b9f382a588160471d52dbbc146de30f1d21a1796861579bfddc5b272d68a4"}
Sep 04 23:59:37.619300 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.619278    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs" event={"ID":"22cc7e55-5fbd-4ddf-89a7-0669d160ffa7","Type":"ContainerDied","Data":"bcbbbb36d62f63fe4124ffcd354b05b0d48cf20abc7ba0a8df964cd108ca4ed5"}
Sep 04 23:59:37.626695 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.626680    2557 scope.go:117] "RemoveContainer" containerID="3fd269d8874b2d9e94eea7328b2e57d8d0bcfccd91868a0e76bc7b8ed43e5570"
Sep 04 23:59:37.626990 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:59:37.626969    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd269d8874b2d9e94eea7328b2e57d8d0bcfccd91868a0e76bc7b8ed43e5570\": container with ID starting with 3fd269d8874b2d9e94eea7328b2e57d8d0bcfccd91868a0e76bc7b8ed43e5570 not found: ID does not exist" containerID="3fd269d8874b2d9e94eea7328b2e57d8d0bcfccd91868a0e76bc7b8ed43e5570"
Sep 04 23:59:37.627091 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.626997    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd269d8874b2d9e94eea7328b2e57d8d0bcfccd91868a0e76bc7b8ed43e5570"} err="failed to get container status \"3fd269d8874b2d9e94eea7328b2e57d8d0bcfccd91868a0e76bc7b8ed43e5570\": rpc error: code = NotFound desc = could not find container \"3fd269d8874b2d9e94eea7328b2e57d8d0bcfccd91868a0e76bc7b8ed43e5570\": container with ID starting with 3fd269d8874b2d9e94eea7328b2e57d8d0bcfccd91868a0e76bc7b8ed43e5570 not found: ID does not exist"
Sep 04 23:59:37.627091 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.627013    2557 scope.go:117] "RemoveContainer" containerID="6e4b9f382a588160471d52dbbc146de30f1d21a1796861579bfddc5b272d68a4"
Sep 04 23:59:37.634925 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.634907    2557 scope.go:117] "RemoveContainer" containerID="6e4b9f382a588160471d52dbbc146de30f1d21a1796861579bfddc5b272d68a4"
Sep 04 23:59:37.635198 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:59:37.635178    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e4b9f382a588160471d52dbbc146de30f1d21a1796861579bfddc5b272d68a4\": container with ID starting with 6e4b9f382a588160471d52dbbc146de30f1d21a1796861579bfddc5b272d68a4 not found: ID does not exist" containerID="6e4b9f382a588160471d52dbbc146de30f1d21a1796861579bfddc5b272d68a4"
Sep 04 23:59:37.635261 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.635205    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e4b9f382a588160471d52dbbc146de30f1d21a1796861579bfddc5b272d68a4"} err="failed to get container status \"6e4b9f382a588160471d52dbbc146de30f1d21a1796861579bfddc5b272d68a4\": rpc error: code = NotFound desc = could not find container \"6e4b9f382a588160471d52dbbc146de30f1d21a1796861579bfddc5b272d68a4\": container with ID starting with 6e4b9f382a588160471d52dbbc146de30f1d21a1796861579bfddc5b272d68a4 not found: ID does not exist"
Sep 04 23:59:37.651966 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.651923    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86ntz\" (UniqueName: \"kubernetes.io/projected/d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af-kube-api-access-86ntz\") pod \"d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af\" (UID: \"d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af\") "
Sep 04 23:59:37.652052 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.652009    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af-plugins-config-volume\") pod \"d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af\" (UID: \"d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af\") "
Sep 04 23:59:37.652099 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.652053    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkkww\" (UniqueName: \"kubernetes.io/projected/22cc7e55-5fbd-4ddf-89a7-0669d160ffa7-kube-api-access-gkkww\") pod \"22cc7e55-5fbd-4ddf-89a7-0669d160ffa7\" (UID: \"22cc7e55-5fbd-4ddf-89a7-0669d160ffa7\") "
Sep 04 23:59:37.652141 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.652099    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22cc7e55-5fbd-4ddf-89a7-0669d160ffa7-plugins-config-volume\") pod \"22cc7e55-5fbd-4ddf-89a7-0669d160ffa7\" (UID: \"22cc7e55-5fbd-4ddf-89a7-0669d160ffa7\") "
Sep 04 23:59:37.652381 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.652355    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af" (UID: "d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:59:37.652541 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.652522    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cc7e55-5fbd-4ddf-89a7-0669d160ffa7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22cc7e55-5fbd-4ddf-89a7-0669d160ffa7" (UID: "22cc7e55-5fbd-4ddf-89a7-0669d160ffa7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:59:37.653971 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.653928    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af-kube-api-access-86ntz" (OuterVolumeSpecName: "kube-api-access-86ntz") pod "d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af" (UID: "d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af"). InnerVolumeSpecName "kube-api-access-86ntz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:59:37.654058 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.653965    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cc7e55-5fbd-4ddf-89a7-0669d160ffa7-kube-api-access-gkkww" (OuterVolumeSpecName: "kube-api-access-gkkww") pod "22cc7e55-5fbd-4ddf-89a7-0669d160ffa7" (UID: "22cc7e55-5fbd-4ddf-89a7-0669d160ffa7"). InnerVolumeSpecName "kube-api-access-gkkww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:59:37.753847 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.753815    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:37.753847 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.753842    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkkww\" (UniqueName: \"kubernetes.io/projected/22cc7e55-5fbd-4ddf-89a7-0669d160ffa7-kube-api-access-gkkww\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:37.753847 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.753852    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22cc7e55-5fbd-4ddf-89a7-0669d160ffa7-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:37.754080 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.753862    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86ntz\" (UniqueName: \"kubernetes.io/projected/d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af-kube-api-access-86ntz\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:37.943610 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.943584    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"]
Sep 04 23:59:37.946049 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.946025    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cff3b-7b9457cd97-44blx"]
Sep 04 23:59:37.955622 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.955598    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"]
Sep 04 23:59:37.958947 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:37.958914    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cff3b-5658fc69db-sddhs"]
Sep 04 23:59:38.079466 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:38.079436    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22cc7e55-5fbd-4ddf-89a7-0669d160ffa7" path="/var/lib/kubelet/pods/22cc7e55-5fbd-4ddf-89a7-0669d160ffa7/volumes"
Sep 04 23:59:38.079874 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:38.079854    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af" path="/var/lib/kubelet/pods/d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af/volumes"
Sep 04 23:59:41.314079 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:41.314047    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 04 23:59:43.985646 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.985608    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4"]
Sep 04 23:59:43.986035 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.986006    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af" containerName="payload-processing"
Sep 04 23:59:43.986035 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.986020    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af" containerName="payload-processing"
Sep 04 23:59:43.986126 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.986042    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22cc7e55-5fbd-4ddf-89a7-0669d160ffa7" containerName="payload-pre-processing"
Sep 04 23:59:43.986126 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.986048    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="22cc7e55-5fbd-4ddf-89a7-0669d160ffa7" containerName="payload-pre-processing"
Sep 04 23:59:43.986126 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.986057    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerName="istio-proxy"
Sep 04 23:59:43.986126 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.986064    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerName="istio-proxy"
Sep 04 23:59:43.986126 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.986076    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerName="istio-proxy"
Sep 04 23:59:43.986126 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.986081    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerName="istio-proxy"
Sep 04 23:59:43.986332 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.986144    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerName="istio-proxy"
Sep 04 23:59:43.986332 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.986152    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="22cc7e55-5fbd-4ddf-89a7-0669d160ffa7" containerName="payload-pre-processing"
Sep 04 23:59:43.986332 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.986158    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed6db1e-25f9-45b6-a94d-ff6d4c24362d" containerName="istio-proxy"
Sep 04 23:59:43.986332 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.986164    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e9c16d-1e7b-4b6a-a836-28a3bd3d33af" containerName="payload-processing"
Sep 04 23:59:43.990476 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.990460    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4"
Sep 04 23:59:43.997025 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.993446    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 23:59:43.997025 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.993454    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k4f64\""
Sep 04 23:59:43.999045 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:43.999019    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4"]
Sep 04 23:59:44.109264 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:44.109231    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4\" (UID: \"a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4"
Sep 04 23:59:44.210011 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:44.209976    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4\" (UID: \"a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4"
Sep 04 23:59:44.210576 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:44.210556    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4\" (UID: \"a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4"
Sep 04 23:59:44.311451 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:44.311384    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4"
Sep 04 23:59:44.581968 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:44.581930    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4"]
Sep 04 23:59:44.584361 ip-10-0-142-236 kubenswrapper[2557]: W0904 23:59:44.584321    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ad1bf9_a9e0_47a7_9cdb_2e79fb2abfa8.slice/crio-67d127ee6ca58dfd0ac1cd8004c85a7d4db1dfb99f1ccecee01fa2a7d9653b26 WatchSource:0}: Error finding container 67d127ee6ca58dfd0ac1cd8004c85a7d4db1dfb99f1ccecee01fa2a7d9653b26: Status 404 returned error can't find the container with id 67d127ee6ca58dfd0ac1cd8004c85a7d4db1dfb99f1ccecee01fa2a7d9653b26
Sep 04 23:59:44.666856 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:44.666829    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4" event={"ID":"a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8","Type":"ContainerStarted","Data":"67d127ee6ca58dfd0ac1cd8004c85a7d4db1dfb99f1ccecee01fa2a7d9653b26"}
Sep 04 23:59:45.673444 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:45.673413    2557 generic.go:356] "Generic (PLEG): container finished" podID="a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8" containerID="e991389f7e195919ea1a27aac21adc7b9cd2a0ae74320472da341efa456e208d" exitCode=0
Sep 04 23:59:45.673834 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:45.673452    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4" event={"ID":"a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8","Type":"ContainerDied","Data":"e991389f7e195919ea1a27aac21adc7b9cd2a0ae74320472da341efa456e208d"}
Sep 04 23:59:46.804451 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:46.804428    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4"
Sep 04 23:59:46.935496 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:46.935419    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8-maas-api-service-ca\") pod \"a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8\" (UID: \"a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8\") "
Sep 04 23:59:46.935755 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:46.935734    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8" (UID: "a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:59:47.036362 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:47.036331    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8-maas-api-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:47.686025 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:47.685996    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4"
Sep 04 23:59:47.686189 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:47.686028    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4" event={"ID":"a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8","Type":"ContainerDied","Data":"67d127ee6ca58dfd0ac1cd8004c85a7d4db1dfb99f1ccecee01fa2a7d9653b26"}
Sep 04 23:59:47.686189 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:47.686075    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67d127ee6ca58dfd0ac1cd8004c85a7d4db1dfb99f1ccecee01fa2a7d9653b26"
Sep 04 23:59:54.445197 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.445160    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"]
Sep 04 23:59:54.445641 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.445459    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965" podUID="78b20687-9123-40e3-95ab-6caa7aa51c28" containerName="payload-pre-processing" containerID="cri-o://888e5aadf5e5851b5e8804fbbd21045c664604a1ca8694d9c1acc2fbcae73f28" gracePeriod=30
Sep 04 23:59:54.583697 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.583664    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4"]
Sep 04 23:59:54.586089 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.586060    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e35897-qnrn4"]
Sep 04 23:59:54.693246 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.693221    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"
Sep 04 23:59:54.732206 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.732177    2557 generic.go:356] "Generic (PLEG): container finished" podID="78b20687-9123-40e3-95ab-6caa7aa51c28" containerID="888e5aadf5e5851b5e8804fbbd21045c664604a1ca8694d9c1acc2fbcae73f28" exitCode=0
Sep 04 23:59:54.732365 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.732222    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"
Sep 04 23:59:54.732365 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.732259    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965" event={"ID":"78b20687-9123-40e3-95ab-6caa7aa51c28","Type":"ContainerDied","Data":"888e5aadf5e5851b5e8804fbbd21045c664604a1ca8694d9c1acc2fbcae73f28"}
Sep 04 23:59:54.732365 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.732312    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965" event={"ID":"78b20687-9123-40e3-95ab-6caa7aa51c28","Type":"ContainerDied","Data":"7936d608fe8e36f2dc9b95aff154926a38bd0d979b2880519d0bdc11b863b91b"}
Sep 04 23:59:54.732365 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.732340    2557 scope.go:117] "RemoveContainer" containerID="888e5aadf5e5851b5e8804fbbd21045c664604a1ca8694d9c1acc2fbcae73f28"
Sep 04 23:59:54.741805 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.741789    2557 scope.go:117] "RemoveContainer" containerID="888e5aadf5e5851b5e8804fbbd21045c664604a1ca8694d9c1acc2fbcae73f28"
Sep 04 23:59:54.742132 ip-10-0-142-236 kubenswrapper[2557]: E0904 23:59:54.742101    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"888e5aadf5e5851b5e8804fbbd21045c664604a1ca8694d9c1acc2fbcae73f28\": container with ID starting with 888e5aadf5e5851b5e8804fbbd21045c664604a1ca8694d9c1acc2fbcae73f28 not found: ID does not exist" containerID="888e5aadf5e5851b5e8804fbbd21045c664604a1ca8694d9c1acc2fbcae73f28"
Sep 04 23:59:54.742222 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.742141    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888e5aadf5e5851b5e8804fbbd21045c664604a1ca8694d9c1acc2fbcae73f28"} err="failed to get container status \"888e5aadf5e5851b5e8804fbbd21045c664604a1ca8694d9c1acc2fbcae73f28\": rpc error: code = NotFound desc = could not find container \"888e5aadf5e5851b5e8804fbbd21045c664604a1ca8694d9c1acc2fbcae73f28\": container with ID starting with 888e5aadf5e5851b5e8804fbbd21045c664604a1ca8694d9c1acc2fbcae73f28 not found: ID does not exist"
Sep 04 23:59:54.799752 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.799719    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l7nb\" (UniqueName: \"kubernetes.io/projected/78b20687-9123-40e3-95ab-6caa7aa51c28-kube-api-access-8l7nb\") pod \"78b20687-9123-40e3-95ab-6caa7aa51c28\" (UID: \"78b20687-9123-40e3-95ab-6caa7aa51c28\") "
Sep 04 23:59:54.799907 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.799783    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b20687-9123-40e3-95ab-6caa7aa51c28-plugins-config-volume\") pod \"78b20687-9123-40e3-95ab-6caa7aa51c28\" (UID: \"78b20687-9123-40e3-95ab-6caa7aa51c28\") "
Sep 04 23:59:54.800178 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.800156    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b20687-9123-40e3-95ab-6caa7aa51c28-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78b20687-9123-40e3-95ab-6caa7aa51c28" (UID: "78b20687-9123-40e3-95ab-6caa7aa51c28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:59:54.801908 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.801883    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b20687-9123-40e3-95ab-6caa7aa51c28-kube-api-access-8l7nb" (OuterVolumeSpecName: "kube-api-access-8l7nb") pod "78b20687-9123-40e3-95ab-6caa7aa51c28" (UID: "78b20687-9123-40e3-95ab-6caa7aa51c28"). InnerVolumeSpecName "kube-api-access-8l7nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:59:54.901135 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.901107    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b20687-9123-40e3-95ab-6caa7aa51c28-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:54.901135 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:54.901129    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l7nb\" (UniqueName: \"kubernetes.io/projected/78b20687-9123-40e3-95ab-6caa7aa51c28-kube-api-access-8l7nb\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 04 23:59:55.054101 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:55.054072    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"]
Sep 04 23:59:55.061144 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:55.061114    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e35897-7bcb865898-cx965"]
Sep 04 23:59:56.079295 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:56.079260    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78b20687-9123-40e3-95ab-6caa7aa51c28" path="/var/lib/kubelet/pods/78b20687-9123-40e3-95ab-6caa7aa51c28/volumes"
Sep 04 23:59:56.079650 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:56.079586    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8" path="/var/lib/kubelet/pods/a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8/volumes"
Sep 04 23:59:57.726980 ip-10-0-142-236 kubenswrapper[2557]: I0904 23:59:57.726950    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:00:00.135158 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.135122    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-88828d-29809440-mhcpl"]
Sep 05 00:00:00.135726 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.135706    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8" containerName="revoke-keys"
Sep 05 00:00:00.135726 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.135727    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8" containerName="revoke-keys"
Sep 05 00:00:00.135894 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.135761    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78b20687-9123-40e3-95ab-6caa7aa51c28" containerName="payload-pre-processing"
Sep 05 00:00:00.135894 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.135769    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="78b20687-9123-40e3-95ab-6caa7aa51c28" containerName="payload-pre-processing"
Sep 05 00:00:00.136037 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.135904    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3ad1bf9-a9e0-47a7-9cdb-2e79fb2abfa8" containerName="revoke-keys"
Sep 05 00:00:00.136037 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.135922    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="78b20687-9123-40e3-95ab-6caa7aa51c28" containerName="payload-pre-processing"
Sep 05 00:00:00.141800 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.141778    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7eb8e-29809440-zqp99"]
Sep 05 00:00:00.141933 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.141889    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-88828d-29809440-mhcpl"
Sep 05 00:00:00.145028 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.145007    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7eb8e-29809440-zqp99"
Sep 05 00:00:00.145215 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.145191    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k4f64\""
Sep 05 00:00:00.148781 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.148761    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-88828d-29809440-mhcpl"]
Sep 05 00:00:00.152526 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.152501    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7eb8e-29809440-zqp99"]
Sep 05 00:00:00.170798 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.170764    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-88828d-29809440-mhcpl"
Sep 05 00:00:00.179973 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.179929    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7eb8e-29809440-zqp99"
Sep 05 00:00:00.226294 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.226118    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29809440-tfddf"]
Sep 05 00:00:00.237293 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.236791    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29809440-tfddf"]
Sep 05 00:00:00.237293 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.236928    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29809440-tfddf"
Sep 05 00:00:00.241705 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.240514    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-np7qn\""
Sep 05 00:00:00.241705 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.240783    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\""
Sep 05 00:00:00.350621 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.350585    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqrsr\" (UniqueName: \"kubernetes.io/projected/40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed-kube-api-access-kqrsr\") pod \"image-pruner-29809440-tfddf\" (UID: \"40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed\") " pod="openshift-image-registry/image-pruner-29809440-tfddf"
Sep 05 00:00:00.350763 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.350676    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed-serviceca\") pod \"image-pruner-29809440-tfddf\" (UID: \"40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed\") " pod="openshift-image-registry/image-pruner-29809440-tfddf"
Sep 05 00:00:00.452067 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.452039    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqrsr\" (UniqueName: \"kubernetes.io/projected/40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed-kube-api-access-kqrsr\") pod \"image-pruner-29809440-tfddf\" (UID: \"40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed\") " pod="openshift-image-registry/image-pruner-29809440-tfddf"
Sep 05 00:00:00.452189 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.452106    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed-serviceca\") pod \"image-pruner-29809440-tfddf\" (UID: \"40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed\") " pod="openshift-image-registry/image-pruner-29809440-tfddf"
Sep 05 00:00:00.452660 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.452640    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed-serviceca\") pod \"image-pruner-29809440-tfddf\" (UID: \"40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed\") " pod="openshift-image-registry/image-pruner-29809440-tfddf"
Sep 05 00:00:00.464503 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.464459    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqrsr\" (UniqueName: \"kubernetes.io/projected/40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed-kube-api-access-kqrsr\") pod \"image-pruner-29809440-tfddf\" (UID: \"40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed\") " pod="openshift-image-registry/image-pruner-29809440-tfddf"
Sep 05 00:00:00.464644 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.464505    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7eb8e-29809440-zqp99"]
Sep 05 00:00:00.469013 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.468678    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-88828d-29809440-mhcpl"]
Sep 05 00:00:00.471273 ip-10-0-142-236 kubenswrapper[2557]: W0905 00:00:00.471250    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619f348d_1041_4016_b884_c1ed013c8a97.slice/crio-706c191465ca630d794ca6ca174d42d8eca9c32f54ded1281b2218643d750104 WatchSource:0}: Error finding container 706c191465ca630d794ca6ca174d42d8eca9c32f54ded1281b2218643d750104: Status 404 returned error can't find the container with id 706c191465ca630d794ca6ca174d42d8eca9c32f54ded1281b2218643d750104
Sep 05 00:00:00.554731 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.554708    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29809440-tfddf"
Sep 05 00:00:00.768538 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.768456    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7eb8e-29809440-zqp99" event={"ID":"848b7885-7763-49c6-9fa7-941c8fdc59db","Type":"ContainerStarted","Data":"1f14a8fb97540fe9339023723dfdf9a9589c8cb86b548cdba14cae373a5e47e8"}
Sep 05 00:00:00.769743 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.769708    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-88828d-29809440-mhcpl" event={"ID":"619f348d-1041-4016-b884-c1ed013c8a97","Type":"ContainerStarted","Data":"706c191465ca630d794ca6ca174d42d8eca9c32f54ded1281b2218643d750104"}
Sep 05 00:00:00.828003 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.827980    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29809440-tfddf"]
Sep 05 00:00:00.828833 ip-10-0-142-236 kubenswrapper[2557]: W0905 00:00:00.828816    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40895fc7_36d0_424f_bcf9_0d1c5c5eb7ed.slice/crio-3b67efc5d57d69f592e2a942924f0f35574401d7ab76f2faf5fb8c9cc0c734cb WatchSource:0}: Error finding container 3b67efc5d57d69f592e2a942924f0f35574401d7ab76f2faf5fb8c9cc0c734cb: Status 404 returned error can't find the container with id 3b67efc5d57d69f592e2a942924f0f35574401d7ab76f2faf5fb8c9cc0c734cb
Sep 05 00:00:00.905072 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.905044    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"]
Sep 05 00:00:00.905294 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:00.905271    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" podUID="421752b2-6564-4051-bc67-c34348a27f6a" containerName="istio-proxy" containerID="cri-o://ac365edca550b65f466d9636f510ef4a99a3594f591c87499569d2f44a984718" gracePeriod=30
Sep 05 00:00:01.777126 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:01.777032    2557 generic.go:356] "Generic (PLEG): container finished" podID="619f348d-1041-4016-b884-c1ed013c8a97" containerID="c46ede07f728ff2ed0ccbe2fe4685b7f03b201e108c10933744c5fecf1c0b727" exitCode=0
Sep 05 00:00:01.777502 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:01.777125    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-88828d-29809440-mhcpl" event={"ID":"619f348d-1041-4016-b884-c1ed013c8a97","Type":"ContainerDied","Data":"c46ede07f728ff2ed0ccbe2fe4685b7f03b201e108c10933744c5fecf1c0b727"}
Sep 05 00:00:01.778564 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:01.778537    2557 generic.go:356] "Generic (PLEG): container finished" podID="848b7885-7763-49c6-9fa7-941c8fdc59db" containerID="75164d36fe1a0b830cc817ead9d3c152406afb0481e506253f3058c09a4c3207" exitCode=0
Sep 05 00:00:01.778684 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:01.778609    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7eb8e-29809440-zqp99" event={"ID":"848b7885-7763-49c6-9fa7-941c8fdc59db","Type":"ContainerDied","Data":"75164d36fe1a0b830cc817ead9d3c152406afb0481e506253f3058c09a4c3207"}
Sep 05 00:00:01.780090 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:01.780056    2557 generic.go:356] "Generic (PLEG): container finished" podID="40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed" containerID="e435df30128ba6cfafa19f4013e54dbd88d2995893fa49fa90f55842b95d85de" exitCode=0
Sep 05 00:00:01.780181 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:01.780095    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29809440-tfddf" event={"ID":"40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed","Type":"ContainerDied","Data":"e435df30128ba6cfafa19f4013e54dbd88d2995893fa49fa90f55842b95d85de"}
Sep 05 00:00:01.780181 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:01.780127    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29809440-tfddf" event={"ID":"40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed","Type":"ContainerStarted","Data":"3b67efc5d57d69f592e2a942924f0f35574401d7ab76f2faf5fb8c9cc0c734cb"}
Sep 05 00:00:02.950123 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:02.950099    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-88828d-29809440-mhcpl"
Sep 05 00:00:02.969211 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:02.969125    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7eb8e-29809440-zqp99"
Sep 05 00:00:02.972825 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:02.972807    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29809440-tfddf"
Sep 05 00:00:03.079671 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.079601    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqrsr\" (UniqueName: \"kubernetes.io/projected/40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed-kube-api-access-kqrsr\") pod \"40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed\" (UID: \"40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed\") "
Sep 05 00:00:03.079671 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.079639    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed-serviceca\") pod \"40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed\" (UID: \"40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed\") "
Sep 05 00:00:03.080028 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.079998    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed-serviceca" (OuterVolumeSpecName: "serviceca") pod "40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed" (UID: "40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:00:03.081707 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.081687    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed-kube-api-access-kqrsr" (OuterVolumeSpecName: "kube-api-access-kqrsr") pod "40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed" (UID: "40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed"). InnerVolumeSpecName "kube-api-access-kqrsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:00:03.180567 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.180540    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqrsr\" (UniqueName: \"kubernetes.io/projected/40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed-kube-api-access-kqrsr\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:03.180708 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.180622    2557 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed-serviceca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:03.449296 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.449266    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 05 00:00:03.449439 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.449325    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" podUID="421752b2-6564-4051-bc67-c34348a27f6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 00:00:03.792804 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.792724    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7eb8e-29809440-zqp99"
Sep 05 00:00:03.792804 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.792767    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7eb8e-29809440-zqp99" event={"ID":"848b7885-7763-49c6-9fa7-941c8fdc59db","Type":"ContainerDied","Data":"1f14a8fb97540fe9339023723dfdf9a9589c8cb86b548cdba14cae373a5e47e8"}
Sep 05 00:00:03.793036 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.792809    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f14a8fb97540fe9339023723dfdf9a9589c8cb86b548cdba14cae373a5e47e8"
Sep 05 00:00:03.794195 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.794173    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29809440-tfddf"
Sep 05 00:00:03.794286 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.794227    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29809440-tfddf" event={"ID":"40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed","Type":"ContainerDied","Data":"3b67efc5d57d69f592e2a942924f0f35574401d7ab76f2faf5fb8c9cc0c734cb"}
Sep 05 00:00:03.794286 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.794270    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b67efc5d57d69f592e2a942924f0f35574401d7ab76f2faf5fb8c9cc0c734cb"
Sep 05 00:00:03.795740 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.795718    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-88828d-29809440-mhcpl" event={"ID":"619f348d-1041-4016-b884-c1ed013c8a97","Type":"ContainerDied","Data":"706c191465ca630d794ca6ca174d42d8eca9c32f54ded1281b2218643d750104"}
Sep 05 00:00:03.795829 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.795759    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="706c191465ca630d794ca6ca174d42d8eca9c32f54ded1281b2218643d750104"
Sep 05 00:00:03.795829 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:03.795767    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-88828d-29809440-mhcpl"
Sep 05 00:00:04.511067 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:04.511024    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"]
Sep 05 00:00:06.143301 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.143276    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 05 00:00:06.306546 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.306516    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-credential-socket\") pod \"421752b2-6564-4051-bc67-c34348a27f6a\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") "
Sep 05 00:00:06.306705 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.306568    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421752b2-6564-4051-bc67-c34348a27f6a-istio-token\") pod \"421752b2-6564-4051-bc67-c34348a27f6a\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") "
Sep 05 00:00:06.306705 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.306601    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-istio-envoy\") pod \"421752b2-6564-4051-bc67-c34348a27f6a\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") "
Sep 05 00:00:06.306705 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.306636    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-istio-data\") pod \"421752b2-6564-4051-bc67-c34348a27f6a\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") "
Sep 05 00:00:06.306705 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.306661    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-workload-socket\") pod \"421752b2-6564-4051-bc67-c34348a27f6a\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") "
Sep 05 00:00:06.306705 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.306691    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztjkw\" (UniqueName: \"kubernetes.io/projected/421752b2-6564-4051-bc67-c34348a27f6a-kube-api-access-ztjkw\") pod \"421752b2-6564-4051-bc67-c34348a27f6a\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") "
Sep 05 00:00:06.306999 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.306716    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421752b2-6564-4051-bc67-c34348a27f6a-istiod-ca-cert\") pod \"421752b2-6564-4051-bc67-c34348a27f6a\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") "
Sep 05 00:00:06.306999 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.306749    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421752b2-6564-4051-bc67-c34348a27f6a-istio-podinfo\") pod \"421752b2-6564-4051-bc67-c34348a27f6a\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") "
Sep 05 00:00:06.306999 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.306802    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "421752b2-6564-4051-bc67-c34348a27f6a" (UID: "421752b2-6564-4051-bc67-c34348a27f6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:00:06.306999 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.306860    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-workload-certs\") pod \"421752b2-6564-4051-bc67-c34348a27f6a\" (UID: \"421752b2-6564-4051-bc67-c34348a27f6a\") "
Sep 05 00:00:06.306999 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.306929    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-istio-data" (OuterVolumeSpecName: "istio-data") pod "421752b2-6564-4051-bc67-c34348a27f6a" (UID: "421752b2-6564-4051-bc67-c34348a27f6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:00:06.307252 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.307021    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "421752b2-6564-4051-bc67-c34348a27f6a" (UID: "421752b2-6564-4051-bc67-c34348a27f6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:00:06.307252 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.307187    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421752b2-6564-4051-bc67-c34348a27f6a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "421752b2-6564-4051-bc67-c34348a27f6a" (UID: "421752b2-6564-4051-bc67-c34348a27f6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:00:06.307252 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.307192    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "421752b2-6564-4051-bc67-c34348a27f6a" (UID: "421752b2-6564-4051-bc67-c34348a27f6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:00:06.307385 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.307271    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:06.307385 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.307289    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:06.307385 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.307304    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:06.307385 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.307317    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:06.307385 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.307331    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421752b2-6564-4051-bc67-c34348a27f6a-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:06.308879 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.308850    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421752b2-6564-4051-bc67-c34348a27f6a-kube-api-access-ztjkw" (OuterVolumeSpecName: "kube-api-access-ztjkw") pod "421752b2-6564-4051-bc67-c34348a27f6a" (UID: "421752b2-6564-4051-bc67-c34348a27f6a"). InnerVolumeSpecName "kube-api-access-ztjkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:00:06.309008 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.308985    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421752b2-6564-4051-bc67-c34348a27f6a-istio-token" (OuterVolumeSpecName: "istio-token") pod "421752b2-6564-4051-bc67-c34348a27f6a" (UID: "421752b2-6564-4051-bc67-c34348a27f6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:00:06.309197 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.309175    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "421752b2-6564-4051-bc67-c34348a27f6a" (UID: "421752b2-6564-4051-bc67-c34348a27f6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:00:06.309197 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.309183    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/421752b2-6564-4051-bc67-c34348a27f6a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "421752b2-6564-4051-bc67-c34348a27f6a" (UID: "421752b2-6564-4051-bc67-c34348a27f6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 00:00:06.407903 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.407874    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztjkw\" (UniqueName: \"kubernetes.io/projected/421752b2-6564-4051-bc67-c34348a27f6a-kube-api-access-ztjkw\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:06.407903 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.407897    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421752b2-6564-4051-bc67-c34348a27f6a-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:06.407903 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.407906    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421752b2-6564-4051-bc67-c34348a27f6a-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:06.408102 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.407915    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421752b2-6564-4051-bc67-c34348a27f6a-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:06.818168 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.818129    2557 generic.go:356] "Generic (PLEG): container finished" podID="421752b2-6564-4051-bc67-c34348a27f6a" containerID="ac365edca550b65f466d9636f510ef4a99a3594f591c87499569d2f44a984718" exitCode=0
Sep 05 00:00:06.818366 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.818166    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" event={"ID":"421752b2-6564-4051-bc67-c34348a27f6a","Type":"ContainerDied","Data":"ac365edca550b65f466d9636f510ef4a99a3594f591c87499569d2f44a984718"}
Sep 05 00:00:06.818452 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.818434    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8" event={"ID":"421752b2-6564-4051-bc67-c34348a27f6a","Type":"ContainerDied","Data":"38203ac6bcdaaad3fa0629d9fddf1724d4320d65ababba7c65993f7b5d513890"}
Sep 05 00:00:06.818507 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.818202    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"
Sep 05 00:00:06.818545 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.818500    2557 scope.go:117] "RemoveContainer" containerID="ac365edca550b65f466d9636f510ef4a99a3594f591c87499569d2f44a984718"
Sep 05 00:00:06.830242 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.829448    2557 scope.go:117] "RemoveContainer" containerID="a0c8043ba841694944b49d0f9e9ebe8de127b5ddfe0f125719eaca1715c8a18c"
Sep 05 00:00:06.840377 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.840357    2557 scope.go:117] "RemoveContainer" containerID="ac365edca550b65f466d9636f510ef4a99a3594f591c87499569d2f44a984718"
Sep 05 00:00:06.840684 ip-10-0-142-236 kubenswrapper[2557]: E0905 00:00:06.840658    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac365edca550b65f466d9636f510ef4a99a3594f591c87499569d2f44a984718\": container with ID starting with ac365edca550b65f466d9636f510ef4a99a3594f591c87499569d2f44a984718 not found: ID does not exist" containerID="ac365edca550b65f466d9636f510ef4a99a3594f591c87499569d2f44a984718"
Sep 05 00:00:06.840770 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.840693    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac365edca550b65f466d9636f510ef4a99a3594f591c87499569d2f44a984718"} err="failed to get container status \"ac365edca550b65f466d9636f510ef4a99a3594f591c87499569d2f44a984718\": rpc error: code = NotFound desc = could not find container \"ac365edca550b65f466d9636f510ef4a99a3594f591c87499569d2f44a984718\": container with ID starting with ac365edca550b65f466d9636f510ef4a99a3594f591c87499569d2f44a984718 not found: ID does not exist"
Sep 05 00:00:06.840770 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.840715    2557 scope.go:117] "RemoveContainer" containerID="a0c8043ba841694944b49d0f9e9ebe8de127b5ddfe0f125719eaca1715c8a18c"
Sep 05 00:00:06.841012 ip-10-0-142-236 kubenswrapper[2557]: E0905 00:00:06.840993    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c8043ba841694944b49d0f9e9ebe8de127b5ddfe0f125719eaca1715c8a18c\": container with ID starting with a0c8043ba841694944b49d0f9e9ebe8de127b5ddfe0f125719eaca1715c8a18c not found: ID does not exist" containerID="a0c8043ba841694944b49d0f9e9ebe8de127b5ddfe0f125719eaca1715c8a18c"
Sep 05 00:00:06.841055 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.841019    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c8043ba841694944b49d0f9e9ebe8de127b5ddfe0f125719eaca1715c8a18c"} err="failed to get container status \"a0c8043ba841694944b49d0f9e9ebe8de127b5ddfe0f125719eaca1715c8a18c\": rpc error: code = NotFound desc = could not find container \"a0c8043ba841694944b49d0f9e9ebe8de127b5ddfe0f125719eaca1715c8a18c\": container with ID starting with a0c8043ba841694944b49d0f9e9ebe8de127b5ddfe0f125719eaca1715c8a18c not found: ID does not exist"
Sep 05 00:00:06.841485 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.841462    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"]
Sep 05 00:00:06.844648 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:06.844626    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e35897-openshift-default-6b65cb4b76-l8sj8"]
Sep 05 00:00:08.079379 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:08.079347    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="421752b2-6564-4051-bc67-c34348a27f6a" path="/var/lib/kubelet/pods/421752b2-6564-4051-bc67-c34348a27f6a/volumes"
Sep 05 00:00:10.605617 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.605581    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"]
Sep 05 00:00:10.606131 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.606033    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619f348d-1041-4016-b884-c1ed013c8a97" containerName="cleanup"
Sep 05 00:00:10.606131 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.606049    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="619f348d-1041-4016-b884-c1ed013c8a97" containerName="cleanup"
Sep 05 00:00:10.606131 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.606064    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="848b7885-7763-49c6-9fa7-941c8fdc59db" containerName="cleanup"
Sep 05 00:00:10.606131 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.606070    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="848b7885-7763-49c6-9fa7-941c8fdc59db" containerName="cleanup"
Sep 05 00:00:10.606131 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.606086    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="421752b2-6564-4051-bc67-c34348a27f6a" containerName="istio-proxy"
Sep 05 00:00:10.606131 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.606092    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="421752b2-6564-4051-bc67-c34348a27f6a" containerName="istio-proxy"
Sep 05 00:00:10.606131 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.606099    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed" containerName="image-pruner"
Sep 05 00:00:10.606131 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.606104    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed" containerName="image-pruner"
Sep 05 00:00:10.606131 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.606112    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="421752b2-6564-4051-bc67-c34348a27f6a" containerName="istio-proxy"
Sep 05 00:00:10.606131 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.606117    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="421752b2-6564-4051-bc67-c34348a27f6a" containerName="istio-proxy"
Sep 05 00:00:10.606459 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.606173    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="848b7885-7763-49c6-9fa7-941c8fdc59db" containerName="cleanup"
Sep 05 00:00:10.606459 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.606182    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="421752b2-6564-4051-bc67-c34348a27f6a" containerName="istio-proxy"
Sep 05 00:00:10.606459 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.606187    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="619f348d-1041-4016-b884-c1ed013c8a97" containerName="cleanup"
Sep 05 00:00:10.606459 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.606193    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed" containerName="image-pruner"
Sep 05 00:00:10.606459 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.606202    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="421752b2-6564-4051-bc67-c34348a27f6a" containerName="istio-proxy"
Sep 05 00:00:10.610546 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.610527    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"
Sep 05 00:00:10.614161 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.614142    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bea561-dockercfg-v8wl4\""
Sep 05 00:00:10.614592 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.614575    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bea561\""
Sep 05 00:00:10.622410 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.622391    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"]
Sep 05 00:00:10.644441 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.644419    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"]
Sep 05 00:00:10.647989 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.647969    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"
Sep 05 00:00:10.658050 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.658030    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"]
Sep 05 00:00:10.744254 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.744220    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn6z8\" (UniqueName: \"kubernetes.io/projected/c22400c6-48cb-45d0-bbc3-2ed640e41a40-kube-api-access-mn6z8\") pod \"payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz\" (UID: \"c22400c6-48cb-45d0-bbc3-2ed640e41a40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"
Sep 05 00:00:10.744432 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.744337    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22400c6-48cb-45d0-bbc3-2ed640e41a40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz\" (UID: \"c22400c6-48cb-45d0-bbc3-2ed640e41a40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"
Sep 05 00:00:10.845259 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.845231    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlfbl\" (UniqueName: \"kubernetes.io/projected/095c5896-2de9-4c6d-8dfa-fcc215b3972d-kube-api-access-dlfbl\") pod \"payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr\" (UID: \"095c5896-2de9-4c6d-8dfa-fcc215b3972d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"
Sep 05 00:00:10.845422 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.845287    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22400c6-48cb-45d0-bbc3-2ed640e41a40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz\" (UID: \"c22400c6-48cb-45d0-bbc3-2ed640e41a40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"
Sep 05 00:00:10.845422 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.845354    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn6z8\" (UniqueName: \"kubernetes.io/projected/c22400c6-48cb-45d0-bbc3-2ed640e41a40-kube-api-access-mn6z8\") pod \"payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz\" (UID: \"c22400c6-48cb-45d0-bbc3-2ed640e41a40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"
Sep 05 00:00:10.845559 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.845431    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095c5896-2de9-4c6d-8dfa-fcc215b3972d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr\" (UID: \"095c5896-2de9-4c6d-8dfa-fcc215b3972d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"
Sep 05 00:00:10.845872 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.845855    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22400c6-48cb-45d0-bbc3-2ed640e41a40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz\" (UID: \"c22400c6-48cb-45d0-bbc3-2ed640e41a40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"
Sep 05 00:00:10.853233 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.853215    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn6z8\" (UniqueName: \"kubernetes.io/projected/c22400c6-48cb-45d0-bbc3-2ed640e41a40-kube-api-access-mn6z8\") pod \"payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz\" (UID: \"c22400c6-48cb-45d0-bbc3-2ed640e41a40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"
Sep 05 00:00:10.921664 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.921606    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"
Sep 05 00:00:10.946126 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.946095    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095c5896-2de9-4c6d-8dfa-fcc215b3972d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr\" (UID: \"095c5896-2de9-4c6d-8dfa-fcc215b3972d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"
Sep 05 00:00:10.946267 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.946148    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlfbl\" (UniqueName: \"kubernetes.io/projected/095c5896-2de9-4c6d-8dfa-fcc215b3972d-kube-api-access-dlfbl\") pod \"payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr\" (UID: \"095c5896-2de9-4c6d-8dfa-fcc215b3972d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"
Sep 05 00:00:10.946726 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.946704    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095c5896-2de9-4c6d-8dfa-fcc215b3972d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr\" (UID: \"095c5896-2de9-4c6d-8dfa-fcc215b3972d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"
Sep 05 00:00:10.953541 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.953515    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlfbl\" (UniqueName: \"kubernetes.io/projected/095c5896-2de9-4c6d-8dfa-fcc215b3972d-kube-api-access-dlfbl\") pod \"payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr\" (UID: \"095c5896-2de9-4c6d-8dfa-fcc215b3972d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"
Sep 05 00:00:10.959469 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:10.959364    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"
Sep 05 00:00:11.195270 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:11.195243    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"]
Sep 05 00:00:11.196443 ip-10-0-142-236 kubenswrapper[2557]: W0905 00:00:11.196419    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22400c6_48cb_45d0_bbc3_2ed640e41a40.slice/crio-8d7794be5a5711e7102f02b6b814423da284399481b988951fd916d3d1c4545b WatchSource:0}: Error finding container 8d7794be5a5711e7102f02b6b814423da284399481b988951fd916d3d1c4545b: Status 404 returned error can't find the container with id 8d7794be5a5711e7102f02b6b814423da284399481b988951fd916d3d1c4545b
Sep 05 00:00:11.238744 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:11.238722    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"]
Sep 05 00:00:11.239373 ip-10-0-142-236 kubenswrapper[2557]: W0905 00:00:11.239350    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095c5896_2de9_4c6d_8dfa_fcc215b3972d.slice/crio-7df91d5065552014089ed8446bdb44054782dafff5c1691b098206d15ba1e363 WatchSource:0}: Error finding container 7df91d5065552014089ed8446bdb44054782dafff5c1691b098206d15ba1e363: Status 404 returned error can't find the container with id 7df91d5065552014089ed8446bdb44054782dafff5c1691b098206d15ba1e363
Sep 05 00:00:11.502080 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:11.502046    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7eb8e-29809440-zqp99"]
Sep 05 00:00:11.507080 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:11.507052    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7eb8e-29809440-zqp99"]
Sep 05 00:00:11.849369 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:11.849269    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr" event={"ID":"095c5896-2de9-4c6d-8dfa-fcc215b3972d","Type":"ContainerStarted","Data":"7e6c688ebe589533b967986aae1bc21241c4def6d5399008f2b1d63f188e076a"}
Sep 05 00:00:11.849810 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:11.849370    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr" event={"ID":"095c5896-2de9-4c6d-8dfa-fcc215b3972d","Type":"ContainerStarted","Data":"7df91d5065552014089ed8446bdb44054782dafff5c1691b098206d15ba1e363"}
Sep 05 00:00:11.849810 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:11.849493    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"
Sep 05 00:00:11.851041 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:11.851013    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz" event={"ID":"c22400c6-48cb-45d0-bbc3-2ed640e41a40","Type":"ContainerStarted","Data":"bfad216dc22a3d84242a3e5cf78abbb63214210b550a9eeb688169898173ab9f"}
Sep 05 00:00:11.851182 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:11.851055    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz" event={"ID":"c22400c6-48cb-45d0-bbc3-2ed640e41a40","Type":"ContainerStarted","Data":"8d7794be5a5711e7102f02b6b814423da284399481b988951fd916d3d1c4545b"}
Sep 05 00:00:11.851182 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:11.851087    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"
Sep 05 00:00:11.867009 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:11.866927    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr" podStartSLOduration=1.866914134 podStartE2EDuration="1.866914134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 00:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 00:00:11.865556547 +0000 UTC m=+1394.283374884" watchObservedRunningTime="2026-09-05 00:00:11.866914134 +0000 UTC m=+1394.284732452"
Sep 05 00:00:11.883413 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:11.883371    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz" podStartSLOduration=1.883357917 podStartE2EDuration="1.883357917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 00:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 00:00:11.880582188 +0000 UTC m=+1394.298400506" watchObservedRunningTime="2026-09-05 00:00:11.883357917 +0000 UTC m=+1394.301176235"
Sep 05 00:00:12.080126 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:12.080096    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="848b7885-7763-49c6-9fa7-941c8fdc59db" path="/var/lib/kubelet/pods/848b7885-7763-49c6-9fa7-941c8fdc59db/volumes"
Sep 05 00:00:14.023732 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:14.023697    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:00:18.019349 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:18.019311    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"]
Sep 05 00:00:18.019746 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:18.019612    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq" podUID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerName="istio-proxy" containerID="cri-o://91a3483df2dabd764bc52c7060f1189cd1a6a2a1517fe63678c4333c0deb6dc7" gracePeriod=30
Sep 05 00:00:19.533019 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.532974    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2" podUID="0f12385e-73c9-4f0d-a5e7-82e3ce8cf675" containerName="main" containerID="cri-o://3cc20d628238019164b775eba275c505da4a7c471f795d53ff1a9d1eef56cc19" gracePeriod=45
Sep 05 00:00:19.772838 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.772816    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 05 00:00:19.819666 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.819602    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-dshm\") pod \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") "
Sep 05 00:00:19.819666 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.819649    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-home\") pod \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") "
Sep 05 00:00:19.819853 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.819674    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-tmp-dir\") pod \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") "
Sep 05 00:00:19.819853 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.819720    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-model-cache\") pod \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") "
Sep 05 00:00:19.819853 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.819786    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjjj9\" (UniqueName: \"kubernetes.io/projected/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-kube-api-access-wjjj9\") pod \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") "
Sep 05 00:00:19.819853 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.819819    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-tls-certs\") pod \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\" (UID: \"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675\") "
Sep 05 00:00:19.820117 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.819958    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f12385e-73c9-4f0d-a5e7-82e3ce8cf675" (UID: "0f12385e-73c9-4f0d-a5e7-82e3ce8cf675"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:00:19.820117 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.819984    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-home" (OuterVolumeSpecName: "home") pod "0f12385e-73c9-4f0d-a5e7-82e3ce8cf675" (UID: "0f12385e-73c9-4f0d-a5e7-82e3ce8cf675"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:00:19.820117 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.820070    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f12385e-73c9-4f0d-a5e7-82e3ce8cf675" (UID: "0f12385e-73c9-4f0d-a5e7-82e3ce8cf675"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:00:19.820246 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.820137    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:19.820246 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.820182    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:19.821911 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.821879    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-kube-api-access-wjjj9" (OuterVolumeSpecName: "kube-api-access-wjjj9") pod "0f12385e-73c9-4f0d-a5e7-82e3ce8cf675" (UID: "0f12385e-73c9-4f0d-a5e7-82e3ce8cf675"). InnerVolumeSpecName "kube-api-access-wjjj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:00:19.821911 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.821883    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f12385e-73c9-4f0d-a5e7-82e3ce8cf675" (UID: "0f12385e-73c9-4f0d-a5e7-82e3ce8cf675"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:00:19.822117 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.822032    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-dshm" (OuterVolumeSpecName: "dshm") pod "0f12385e-73c9-4f0d-a5e7-82e3ce8cf675" (UID: "0f12385e-73c9-4f0d-a5e7-82e3ce8cf675"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:00:19.898782 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.898750    2557 generic.go:356] "Generic (PLEG): container finished" podID="0f12385e-73c9-4f0d-a5e7-82e3ce8cf675" containerID="3cc20d628238019164b775eba275c505da4a7c471f795d53ff1a9d1eef56cc19" exitCode=0
Sep 05 00:00:19.898925 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.898797    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"
Sep 05 00:00:19.898925 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.898832    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2" event={"ID":"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675","Type":"ContainerDied","Data":"3cc20d628238019164b775eba275c505da4a7c471f795d53ff1a9d1eef56cc19"}
Sep 05 00:00:19.898925 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.898875    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2" event={"ID":"0f12385e-73c9-4f0d-a5e7-82e3ce8cf675","Type":"ContainerDied","Data":"ed70c1ec1938d2c4047108959c3b54d794597777db0c091718a1a20695dd5285"}
Sep 05 00:00:19.898925 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.898892    2557 scope.go:117] "RemoveContainer" containerID="3cc20d628238019164b775eba275c505da4a7c471f795d53ff1a9d1eef56cc19"
Sep 05 00:00:19.907895 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.907878    2557 scope.go:117] "RemoveContainer" containerID="3cc20d628238019164b775eba275c505da4a7c471f795d53ff1a9d1eef56cc19"
Sep 05 00:00:19.908183 ip-10-0-142-236 kubenswrapper[2557]: E0905 00:00:19.908163    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc20d628238019164b775eba275c505da4a7c471f795d53ff1a9d1eef56cc19\": container with ID starting with 3cc20d628238019164b775eba275c505da4a7c471f795d53ff1a9d1eef56cc19 not found: ID does not exist" containerID="3cc20d628238019164b775eba275c505da4a7c471f795d53ff1a9d1eef56cc19"
Sep 05 00:00:19.908255 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.908194    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc20d628238019164b775eba275c505da4a7c471f795d53ff1a9d1eef56cc19"} err="failed to get container status \"3cc20d628238019164b775eba275c505da4a7c471f795d53ff1a9d1eef56cc19\": rpc error: code = NotFound desc = could not find container \"3cc20d628238019164b775eba275c505da4a7c471f795d53ff1a9d1eef56cc19\": container with ID starting with 3cc20d628238019164b775eba275c505da4a7c471f795d53ff1a9d1eef56cc19 not found: ID does not exist"
Sep 05 00:00:19.919844 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.919823    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"]
Sep 05 00:00:19.921156 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.921137    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:19.921156 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.921158    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:19.921275 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.921166    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:19.921275 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.921174    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjjj9\" (UniqueName: \"kubernetes.io/projected/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675-kube-api-access-wjjj9\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:19.923362 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:19.923343    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-88828d/rate-test-model-88828d-kserve-67bbcf84f9-c2jk2"]
Sep 05 00:00:20.079419 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:20.079345    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f12385e-73c9-4f0d-a5e7-82e3ce8cf675" path="/var/lib/kubelet/pods/0f12385e-73c9-4f0d-a5e7-82e3ce8cf675/volumes"
Sep 05 00:00:22.858115 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:22.858083    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"
Sep 05 00:00:22.858486 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:22.858464    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"
Sep 05 00:00:23.278145 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.278122    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 05 00:00:23.348838 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.348797    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-token\") pod \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") "
Sep 05 00:00:23.349023 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.348859    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-credential-socket\") pod \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") "
Sep 05 00:00:23.349023 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.348912    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-data\") pod \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") "
Sep 05 00:00:23.349023 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.348972    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istiod-ca-cert\") pod \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") "
Sep 05 00:00:23.349023 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.348999    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-workload-socket\") pod \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") "
Sep 05 00:00:23.349247 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.349063    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-workload-certs\") pod \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") "
Sep 05 00:00:23.349247 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.349096    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4flw4\" (UniqueName: \"kubernetes.io/projected/d46a512a-b1f4-4d41-98c1-a044b37afd9c-kube-api-access-4flw4\") pod \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") "
Sep 05 00:00:23.349247 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.349124    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-podinfo\") pod \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") "
Sep 05 00:00:23.349247 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.349155    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-envoy\") pod \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\" (UID: \"d46a512a-b1f4-4d41-98c1-a044b37afd9c\") "
Sep 05 00:00:23.349247 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.349151    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d46a512a-b1f4-4d41-98c1-a044b37afd9c" (UID: "d46a512a-b1f4-4d41-98c1-a044b37afd9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:00:23.349247 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.349211    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-data" (OuterVolumeSpecName: "istio-data") pod "d46a512a-b1f4-4d41-98c1-a044b37afd9c" (UID: "d46a512a-b1f4-4d41-98c1-a044b37afd9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:00:23.349247 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.349227    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d46a512a-b1f4-4d41-98c1-a044b37afd9c" (UID: "d46a512a-b1f4-4d41-98c1-a044b37afd9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:00:23.349599 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.349359    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d46a512a-b1f4-4d41-98c1-a044b37afd9c" (UID: "d46a512a-b1f4-4d41-98c1-a044b37afd9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:00:23.349599 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.349395    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d46a512a-b1f4-4d41-98c1-a044b37afd9c" (UID: "d46a512a-b1f4-4d41-98c1-a044b37afd9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:00:23.349599 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.349528    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:23.349599 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.349557    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:23.349599 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.349573    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:23.349599 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.349587    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:23.349599 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.349599    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:23.351067 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.351046    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-token" (OuterVolumeSpecName: "istio-token") pod "d46a512a-b1f4-4d41-98c1-a044b37afd9c" (UID: "d46a512a-b1f4-4d41-98c1-a044b37afd9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:00:23.351216 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.351195    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d46a512a-b1f4-4d41-98c1-a044b37afd9c" (UID: "d46a512a-b1f4-4d41-98c1-a044b37afd9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 00:00:23.351585 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.351565    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d46a512a-b1f4-4d41-98c1-a044b37afd9c" (UID: "d46a512a-b1f4-4d41-98c1-a044b37afd9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:00:23.351645 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.351585    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46a512a-b1f4-4d41-98c1-a044b37afd9c-kube-api-access-4flw4" (OuterVolumeSpecName: "kube-api-access-4flw4") pod "d46a512a-b1f4-4d41-98c1-a044b37afd9c" (UID: "d46a512a-b1f4-4d41-98c1-a044b37afd9c"). InnerVolumeSpecName "kube-api-access-4flw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:00:23.450293 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.450273    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:23.450379 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.450293    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4flw4\" (UniqueName: \"kubernetes.io/projected/d46a512a-b1f4-4d41-98c1-a044b37afd9c-kube-api-access-4flw4\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:23.450379 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.450303    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:23.450379 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.450311    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46a512a-b1f4-4d41-98c1-a044b37afd9c-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:00:23.922798 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.922707    2557 generic.go:356] "Generic (PLEG): container finished" podID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerID="91a3483df2dabd764bc52c7060f1189cd1a6a2a1517fe63678c4333c0deb6dc7" exitCode=0
Sep 05 00:00:23.922798 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.922767    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"
Sep 05 00:00:23.923339 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.922796    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq" event={"ID":"d46a512a-b1f4-4d41-98c1-a044b37afd9c","Type":"ContainerDied","Data":"91a3483df2dabd764bc52c7060f1189cd1a6a2a1517fe63678c4333c0deb6dc7"}
Sep 05 00:00:23.923339 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.922875    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq" event={"ID":"d46a512a-b1f4-4d41-98c1-a044b37afd9c","Type":"ContainerDied","Data":"66c2e86a8802383705509dd33745f805b09b601470ad08fcb879e0b38cd7028f"}
Sep 05 00:00:23.923339 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.922899    2557 scope.go:117] "RemoveContainer" containerID="91a3483df2dabd764bc52c7060f1189cd1a6a2a1517fe63678c4333c0deb6dc7"
Sep 05 00:00:23.932543 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.932525    2557 scope.go:117] "RemoveContainer" containerID="552bdfe59124b21ae546fcd29c33f700428c4a4c16c088b4a45944f49aa74d79"
Sep 05 00:00:23.940780 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.940759    2557 scope.go:117] "RemoveContainer" containerID="91a3483df2dabd764bc52c7060f1189cd1a6a2a1517fe63678c4333c0deb6dc7"
Sep 05 00:00:23.941051 ip-10-0-142-236 kubenswrapper[2557]: E0905 00:00:23.941035    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a3483df2dabd764bc52c7060f1189cd1a6a2a1517fe63678c4333c0deb6dc7\": container with ID starting with 91a3483df2dabd764bc52c7060f1189cd1a6a2a1517fe63678c4333c0deb6dc7 not found: ID does not exist" containerID="91a3483df2dabd764bc52c7060f1189cd1a6a2a1517fe63678c4333c0deb6dc7"
Sep 05 00:00:23.941111 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.941061    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a3483df2dabd764bc52c7060f1189cd1a6a2a1517fe63678c4333c0deb6dc7"} err="failed to get container status \"91a3483df2dabd764bc52c7060f1189cd1a6a2a1517fe63678c4333c0deb6dc7\": rpc error: code = NotFound desc = could not find container \"91a3483df2dabd764bc52c7060f1189cd1a6a2a1517fe63678c4333c0deb6dc7\": container with ID starting with 91a3483df2dabd764bc52c7060f1189cd1a6a2a1517fe63678c4333c0deb6dc7 not found: ID does not exist"
Sep 05 00:00:23.941111 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.941075    2557 scope.go:117] "RemoveContainer" containerID="552bdfe59124b21ae546fcd29c33f700428c4a4c16c088b4a45944f49aa74d79"
Sep 05 00:00:23.941359 ip-10-0-142-236 kubenswrapper[2557]: E0905 00:00:23.941343    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552bdfe59124b21ae546fcd29c33f700428c4a4c16c088b4a45944f49aa74d79\": container with ID starting with 552bdfe59124b21ae546fcd29c33f700428c4a4c16c088b4a45944f49aa74d79 not found: ID does not exist" containerID="552bdfe59124b21ae546fcd29c33f700428c4a4c16c088b4a45944f49aa74d79"
Sep 05 00:00:23.941405 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.941365    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552bdfe59124b21ae546fcd29c33f700428c4a4c16c088b4a45944f49aa74d79"} err="failed to get container status \"552bdfe59124b21ae546fcd29c33f700428c4a4c16c088b4a45944f49aa74d79\": rpc error: code = NotFound desc = could not find container \"552bdfe59124b21ae546fcd29c33f700428c4a4c16c088b4a45944f49aa74d79\": container with ID starting with 552bdfe59124b21ae546fcd29c33f700428c4a4c16c088b4a45944f49aa74d79 not found: ID does not exist"
Sep 05 00:00:23.944890 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.944868    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"]
Sep 05 00:00:23.948608 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:23.948586    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f7eb8e-openshift-default-64dbd55df8-xmmwq"]
Sep 05 00:00:24.080338 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:24.080308    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" path="/var/lib/kubelet/pods/d46a512a-b1f4-4d41-98c1-a044b37afd9c/volumes"
Sep 05 00:00:34.639495 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.639444    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"]
Sep 05 00:00:34.640037 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.640018    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f12385e-73c9-4f0d-a5e7-82e3ce8cf675" containerName="main"
Sep 05 00:00:34.640037 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.640039    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f12385e-73c9-4f0d-a5e7-82e3ce8cf675" containerName="main"
Sep 05 00:00:34.640187 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.640056    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerName="istio-proxy"
Sep 05 00:00:34.640187 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.640064    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerName="istio-proxy"
Sep 05 00:00:34.640187 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.640088    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerName="istio-proxy"
Sep 05 00:00:34.640187 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.640097    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerName="istio-proxy"
Sep 05 00:00:34.640387 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.640187    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerName="istio-proxy"
Sep 05 00:00:34.640387 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.640200    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f12385e-73c9-4f0d-a5e7-82e3ce8cf675" containerName="main"
Sep 05 00:00:34.640387 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.640210    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d46a512a-b1f4-4d41-98c1-a044b37afd9c" containerName="istio-proxy"
Sep 05 00:00:34.643455 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.643427    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.645990 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.645966    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-64702c-openshift-default-dockercfg-47mk9\""
Sep 05 00:00:34.655838 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.655814    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"]
Sep 05 00:00:34.740692 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.740656    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-credential-socket\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.740882 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.740731    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2b2d\" (UniqueName: \"kubernetes.io/projected/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-kube-api-access-h2b2d\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.740882 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.740770    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istiod-ca-cert\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.740882 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.740796    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-envoy\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.740882 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.740820    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-podinfo\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.740882 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.740871    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-token\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.741196 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.740912    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-workload-certs\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.741196 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.740969    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-data\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.741196 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.741004    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-workload-socket\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.841773 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.841730    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-token\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.841971 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.841784    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-workload-certs\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.841971 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.841810    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-data\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.841971 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.841839    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-workload-socket\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.841971 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.841893    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-credential-socket\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.841971 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.841968    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2b2d\" (UniqueName: \"kubernetes.io/projected/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-kube-api-access-h2b2d\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.842256 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.842003    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istiod-ca-cert\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.842256 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.842029    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-envoy\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.842256 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.842052    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-podinfo\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.842398 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.842274    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-workload-certs\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.842398 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.842321    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-data\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.842398 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.842352    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-workload-socket\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.842558 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.842399    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-credential-socket\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.842746 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.842722    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istiod-ca-cert\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.844351 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.844321    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-envoy\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.844557 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.844540    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-podinfo\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.855890 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.855864    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2b2d\" (UniqueName: \"kubernetes.io/projected/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-kube-api-access-h2b2d\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.856012 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.855997    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-token\") pod \"e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") " pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:34.955048 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:34.955025    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:35.235223 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:35.235196    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"]
Sep 05 00:00:35.237024 ip-10-0-142-236 kubenswrapper[2557]: W0905 00:00:35.236993    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c367d4c_60b9_4fb2_afbe_3fc458fb3a07.slice/crio-116d31de2b9e81e6ed48be2ecf2d5218c2c1aad2de7517e84f5ff601d34ae768 WatchSource:0}: Error finding container 116d31de2b9e81e6ed48be2ecf2d5218c2c1aad2de7517e84f5ff601d34ae768: Status 404 returned error can't find the container with id 116d31de2b9e81e6ed48be2ecf2d5218c2c1aad2de7517e84f5ff601d34ae768
Sep 05 00:00:35.239215 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:35.239181    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 05 00:00:35.239301 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:35.239247    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 05 00:00:35.239301 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:35.239280    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 05 00:00:35.992083 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:35.992048    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv" event={"ID":"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07","Type":"ContainerStarted","Data":"f1ecc175e669688a17e892e4757aac4db3a3c6487c2e52d33f3948321abaddc5"}
Sep 05 00:00:35.992454 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:35.992130    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv" event={"ID":"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07","Type":"ContainerStarted","Data":"116d31de2b9e81e6ed48be2ecf2d5218c2c1aad2de7517e84f5ff601d34ae768"}
Sep 05 00:00:36.013299 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:36.013253    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv" podStartSLOduration=2.013237336 podStartE2EDuration="2.013237336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 00:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 00:00:36.010870674 +0000 UTC m=+1418.428688992" watchObservedRunningTime="2026-09-05 00:00:36.013237336 +0000 UTC m=+1418.431055712"
Sep 05 00:00:36.955872 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:36.955834    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:36.961128 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:36.961104    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:36.998035 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:36.998010    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:36.998858 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:36.998836    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:45.716468 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:45.716436    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:00:57.115002 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:57.114970    2557 generic.go:356] "Generic (PLEG): container finished" podID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" containerID="f1ecc175e669688a17e892e4757aac4db3a3c6487c2e52d33f3948321abaddc5" exitCode=0
Sep 05 00:00:57.115413 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:57.115022    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv" event={"ID":"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07","Type":"ContainerDied","Data":"f1ecc175e669688a17e892e4757aac4db3a3c6487c2e52d33f3948321abaddc5"}
Sep 05 00:00:57.115413 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:57.115307    2557 scope.go:117] "RemoveContainer" containerID="f1ecc175e669688a17e892e4757aac4db3a3c6487c2e52d33f3948321abaddc5"
Sep 05 00:00:57.116306 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:57.116280    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 05 00:00:57.116366 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:57.116346    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 05 00:00:57.116415 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:57.116376    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 05 00:00:58.122407 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:58.122370    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv" event={"ID":"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07","Type":"ContainerStarted","Data":"101d9e9b27416c648c16c106d1eb0cbd39a3934d3c5ef24e8d27b98ac83246d0"}
Sep 05 00:00:58.955402 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:58.955366    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:00:58.955626 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:58.955606    2557 patch_prober.go:28] interesting pod/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Sep 05 00:00:58.955697 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:58.955652    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv" podUID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 05 00:00:59.955734 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:59.955698    2557 patch_prober.go:28] interesting pod/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Sep 05 00:00:59.956205 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:00:59.955757    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv" podUID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 05 00:01:00.956455 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:00.956397    2557 patch_prober.go:28] interesting pod/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Sep 05 00:01:00.956984 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:00.956467    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv" podUID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 05 00:01:01.955782 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:01.955745    2557 patch_prober.go:28] interesting pod/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Sep 05 00:01:01.955976 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:01.955803    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv" podUID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 05 00:01:02.955673 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:02.955641    2557 patch_prober.go:28] interesting pod/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Sep 05 00:01:02.956097 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:02.955696    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv" podUID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 05 00:01:03.959809 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:03.959779    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:01:03.960290 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:03.960173    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:01:03.960864 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:03.960845    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:01:13.421664 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.421627    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"]
Sep 05 00:01:13.426239 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.426222    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.428745 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.428723    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bea561\"/\"kube-root-ca.crt\""
Sep 05 00:01:13.428842 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.428770    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bea561\"/\"openshift-service-ca.crt\""
Sep 05 00:01:13.430126 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.430100    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bea561\"/\"default-dockercfg-vdfcn\""
Sep 05 00:01:13.430236 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.430185    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bea561\"/\"ipp-route-bea561-kserve-self-signed-certs\""
Sep 05 00:01:13.434520 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.434053    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"]
Sep 05 00:01:13.490292 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.490260    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9965b46e-1c91-4f94-b7fe-1ae9303659d9-tls-certs\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.490450 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.490339    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-dshm\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.490450 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.490372    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jl99\" (UniqueName: \"kubernetes.io/projected/9965b46e-1c91-4f94-b7fe-1ae9303659d9-kube-api-access-2jl99\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.490580 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.490488    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-tmp-dir\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.490580 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.490528    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-model-cache\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.490679 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.490582    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-home\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.591670 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.591641    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jl99\" (UniqueName: \"kubernetes.io/projected/9965b46e-1c91-4f94-b7fe-1ae9303659d9-kube-api-access-2jl99\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.591818 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.591691    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-tmp-dir\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.591818 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.591710    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-model-cache\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.591818 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.591752    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-home\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.591818 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.591791    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9965b46e-1c91-4f94-b7fe-1ae9303659d9-tls-certs\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.592066 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.591833    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-dshm\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.592136 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.592116    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-tmp-dir\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.592172 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.592129    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-model-cache\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.592207 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.592185    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-home\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.594021 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.594003    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-dshm\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.594278 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.594259    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9965b46e-1c91-4f94-b7fe-1ae9303659d9-tls-certs\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.598882 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.598861    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jl99\" (UniqueName: \"kubernetes.io/projected/9965b46e-1c91-4f94-b7fe-1ae9303659d9-kube-api-access-2jl99\") pod \"ipp-route-bea561-kserve-6fb7f58787-58zrq\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") " pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:13.736636 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:13.736611    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:14.019294 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:14.019262    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"]
Sep 05 00:01:14.020985 ip-10-0-142-236 kubenswrapper[2557]: W0905 00:01:14.020956    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9965b46e_1c91_4f94_b7fe_1ae9303659d9.slice/crio-90b5255c4d58e9e316afb84db04787df40de48d38131f8b8b5625fbb569dc0eb WatchSource:0}: Error finding container 90b5255c4d58e9e316afb84db04787df40de48d38131f8b8b5625fbb569dc0eb: Status 404 returned error can't find the container with id 90b5255c4d58e9e316afb84db04787df40de48d38131f8b8b5625fbb569dc0eb
Sep 05 00:01:14.214286 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:14.214251    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq" event={"ID":"9965b46e-1c91-4f94-b7fe-1ae9303659d9","Type":"ContainerStarted","Data":"069fcb4a467326080bf59a746193dd14e18cf6fd5342cd8c2084b7257ad9288a"}
Sep 05 00:01:14.214439 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:14.214297    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq" event={"ID":"9965b46e-1c91-4f94-b7fe-1ae9303659d9","Type":"ContainerStarted","Data":"90b5255c4d58e9e316afb84db04787df40de48d38131f8b8b5625fbb569dc0eb"}
Sep 05 00:01:14.229985 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:14.229915    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq" podStartSLOduration=1.229881081 podStartE2EDuration="1.229881081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 00:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 00:01:14.22915007 +0000 UTC m=+1456.646968389" watchObservedRunningTime="2026-09-05 00:01:14.229881081 +0000 UTC m=+1456.647699445"
Sep 05 00:01:17.814217 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:17.814184    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:01:23.737115 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:23.737079    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:23.737515 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:23.737130    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:23.749556 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:23.749529    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:24.284411 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:24.284382    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:01:33.022814 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:33.022784    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:01:40.871528 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:40.871497    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr_095c5896-2de9-4c6d-8dfa-fcc215b3972d/payload-processing/0.log"
Sep 05 00:01:41.702660 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.702630    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"]
Sep 05 00:01:41.706358 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.706340    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.708810 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.708792    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-b3c355-openshift-default-dockercfg-wm682\""
Sep 05 00:01:41.717346 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.717325    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"]
Sep 05 00:01:41.824638 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.824597    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-credential-socket\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.824827 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.824645    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-token\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.824827 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.824703    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-envoy\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.824827 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.824751    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-workload-socket\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.824827 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.824777    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-workload-certs\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.824827 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.824808    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-data\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.825132 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.824848    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54dw9\" (UniqueName: \"kubernetes.io/projected/862597e0-ab54-4b16-b39b-bb1fe1b3811b-kube-api-access-54dw9\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.825132 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.824974    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.825132 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.825018    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-podinfo\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.925647 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.925615    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.926125 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.925650    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-podinfo\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.926125 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.925681    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-credential-socket\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.926125 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.925714    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-token\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.926125 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.925756    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-envoy\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.926125 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.925810    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-workload-socket\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.926125 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.925841    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-workload-certs\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.926125 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.925890    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-data\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.926125 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.925968    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54dw9\" (UniqueName: \"kubernetes.io/projected/862597e0-ab54-4b16-b39b-bb1fe1b3811b-kube-api-access-54dw9\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.926533 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.926161    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-credential-socket\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.926533 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.926244    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-workload-socket\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.926533 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.926343    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-workload-certs\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.926533 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.926381    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-data\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.926533 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.926447    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.928581 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.928563    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-envoy\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.928897 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.928876    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-podinfo\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.933076 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.933054    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-token\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:41.933732 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:41.933710    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54dw9\" (UniqueName: \"kubernetes.io/projected/862597e0-ab54-4b16-b39b-bb1fe1b3811b-kube-api-access-54dw9\") pod \"e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") " pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:42.019147 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:42.019066    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:42.295564 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:42.295536    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"]
Sep 05 00:01:42.296543 ip-10-0-142-236 kubenswrapper[2557]: W0905 00:01:42.296521    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862597e0_ab54_4b16_b39b_bb1fe1b3811b.slice/crio-dba532bc5a6d7ef4c7de395f6c2a163ecb019b1d2d01c0cb5b51fef76b97ae33 WatchSource:0}: Error finding container dba532bc5a6d7ef4c7de395f6c2a163ecb019b1d2d01c0cb5b51fef76b97ae33: Status 404 returned error can't find the container with id dba532bc5a6d7ef4c7de395f6c2a163ecb019b1d2d01c0cb5b51fef76b97ae33
Sep 05 00:01:42.298787 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:42.298756    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 05 00:01:42.298869 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:42.298821    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 05 00:01:42.298869 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:42.298853    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 05 00:01:42.372605 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:42.372573    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq" event={"ID":"862597e0-ab54-4b16-b39b-bb1fe1b3811b","Type":"ContainerStarted","Data":"f5b0fef2593d6988a03726b0168ddaf33e7b3251e3b67cc69db0ef215f1f7d60"}
Sep 05 00:01:42.372702 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:42.372645    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq" event={"ID":"862597e0-ab54-4b16-b39b-bb1fe1b3811b","Type":"ContainerStarted","Data":"dba532bc5a6d7ef4c7de395f6c2a163ecb019b1d2d01c0cb5b51fef76b97ae33"}
Sep 05 00:01:42.391525 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:42.391469    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq" podStartSLOduration=1.391452507 podStartE2EDuration="1.391452507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 00:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 00:01:42.39043221 +0000 UTC m=+1484.808250529" watchObservedRunningTime="2026-09-05 00:01:42.391452507 +0000 UTC m=+1484.809270824"
Sep 05 00:01:43.019676 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:43.019644    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:43.024171 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:43.024145    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:43.378644 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:43.378575    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:43.379494 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:43.379475    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:01:46.185780 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:46.185750    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"]
Sep 05 00:01:46.189437 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:46.189419    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"
Sep 05 00:01:46.191950 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:46.191916    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-b3c355-dockercfg-n57v4\""
Sep 05 00:01:46.192071 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:46.191967    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-b3c355\""
Sep 05 00:01:46.198133 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:46.198110    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"]
Sep 05 00:01:46.265326 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:46.265289    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad411092-f763-4850-b6a9-e854493d2beb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h\" (UID: \"ad411092-f763-4850-b6a9-e854493d2beb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"
Sep 05 00:01:46.265510 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:46.265477    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr7w4\" (UniqueName: \"kubernetes.io/projected/ad411092-f763-4850-b6a9-e854493d2beb-kube-api-access-rr7w4\") pod \"payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h\" (UID: \"ad411092-f763-4850-b6a9-e854493d2beb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"
Sep 05 00:01:46.366093 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:46.366057    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad411092-f763-4850-b6a9-e854493d2beb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h\" (UID: \"ad411092-f763-4850-b6a9-e854493d2beb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"
Sep 05 00:01:46.366303 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:46.366143    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr7w4\" (UniqueName: \"kubernetes.io/projected/ad411092-f763-4850-b6a9-e854493d2beb-kube-api-access-rr7w4\") pod \"payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h\" (UID: \"ad411092-f763-4850-b6a9-e854493d2beb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"
Sep 05 00:01:46.366815 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:46.366787    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad411092-f763-4850-b6a9-e854493d2beb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h\" (UID: \"ad411092-f763-4850-b6a9-e854493d2beb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"
Sep 05 00:01:46.373656 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:46.373631    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr7w4\" (UniqueName: \"kubernetes.io/projected/ad411092-f763-4850-b6a9-e854493d2beb-kube-api-access-rr7w4\") pod \"payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h\" (UID: \"ad411092-f763-4850-b6a9-e854493d2beb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"
Sep 05 00:01:46.501064 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:46.501041    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"
Sep 05 00:01:46.772303 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:46.772275    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"]
Sep 05 00:01:46.773316 ip-10-0-142-236 kubenswrapper[2557]: W0905 00:01:46.773288    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad411092_f763_4850_b6a9_e854493d2beb.slice/crio-2b2b6b4170522f191e0d17732f3b5eaeb3ad6316c9b4b8731e8138f20eba0bf2 WatchSource:0}: Error finding container 2b2b6b4170522f191e0d17732f3b5eaeb3ad6316c9b4b8731e8138f20eba0bf2: Status 404 returned error can't find the container with id 2b2b6b4170522f191e0d17732f3b5eaeb3ad6316c9b4b8731e8138f20eba0bf2
Sep 05 00:01:47.401840 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:47.401803    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h" event={"ID":"ad411092-f763-4850-b6a9-e854493d2beb","Type":"ContainerStarted","Data":"8d62627edef932d322d64c51454e4abf201a324cbcd30e91eadfe75a26271c7a"}
Sep 05 00:01:47.402262 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:47.401872    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"
Sep 05 00:01:47.402262 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:47.401897    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h" event={"ID":"ad411092-f763-4850-b6a9-e854493d2beb","Type":"ContainerStarted","Data":"2b2b6b4170522f191e0d17732f3b5eaeb3ad6316c9b4b8731e8138f20eba0bf2"}
Sep 05 00:01:47.418503 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:47.418455    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h" podStartSLOduration=1.41844058 podStartE2EDuration="1.41844058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 00:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 00:01:47.416826625 +0000 UTC m=+1489.834644945" watchObservedRunningTime="2026-09-05 00:01:47.41844058 +0000 UTC m=+1489.836258899"
Sep 05 00:01:49.521668 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:49.521633    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:01:58.148607 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:58.148580    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 05 00:01:58.151134 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:58.151115    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 05 00:01:58.410222 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:58.410145    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"
Sep 05 00:01:59.338725 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:59.338248    2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": read tcp 10.134.0.2:44220->10.134.0.83:15021: read: connection reset by peer" start-of-body=
Sep 05 00:01:59.338725 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:59.338314    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq" podUID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": read tcp 10.134.0.2:44220->10.134.0.83:15021: read: connection reset by peer"
Sep 05 00:01:59.475802 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:59.475774    2557 generic.go:356] "Generic (PLEG): container finished" podID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" containerID="f5b0fef2593d6988a03726b0168ddaf33e7b3251e3b67cc69db0ef215f1f7d60" exitCode=0
Sep 05 00:01:59.475926 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:59.475851    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq" event={"ID":"862597e0-ab54-4b16-b39b-bb1fe1b3811b","Type":"ContainerDied","Data":"f5b0fef2593d6988a03726b0168ddaf33e7b3251e3b67cc69db0ef215f1f7d60"}
Sep 05 00:01:59.476312 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:59.476295    2557 scope.go:117] "RemoveContainer" containerID="f5b0fef2593d6988a03726b0168ddaf33e7b3251e3b67cc69db0ef215f1f7d60"
Sep 05 00:01:59.477413 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:59.477383    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 05 00:01:59.477505 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:59.477472    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 05 00:01:59.477572 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:01:59.477543    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 05 00:02:00.488072 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:00.488031    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq" event={"ID":"862597e0-ab54-4b16-b39b-bb1fe1b3811b","Type":"ContainerStarted","Data":"0e021a26f3feed0059f4398a348b3268bd1bc6ad6df0b0fc139233b563e61051"}
Sep 05 00:02:01.019452 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:01.019416    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:02:01.019649 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:01.019626    2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Sep 05 00:02:01.019701 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:01.019676    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq" podUID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 05 00:02:02.019595 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:02.019560    2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Sep 05 00:02:02.020002 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:02.019618    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq" podUID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 05 00:02:03.020395 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:03.020358    2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Sep 05 00:02:03.020771 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:03.020444    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq" podUID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 05 00:02:04.020213 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:04.020177    2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Sep 05 00:02:04.020383 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:04.020244    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq" podUID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 05 00:02:05.019714 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:05.019675    2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Sep 05 00:02:05.020211 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:05.019740    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq" podUID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 05 00:02:06.023296 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:06.023267    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:02:06.023778 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:06.023622    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:02:06.024409 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:06.024387    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:02:09.996334 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:09.996300    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7"]
Sep 05 00:02:09.998933 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:09.998917    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7"
Sep 05 00:02:10.001277 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:10.001252    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k4f64\""
Sep 05 00:02:10.001426 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:10.001284    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 00:02:10.006737 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:10.006714    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7"]
Sep 05 00:02:10.094368 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:10.094333    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c922a1-85ce-484d-87cd-ef7b332acf56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7\" (UID: \"80c922a1-85ce-484d-87cd-ef7b332acf56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7"
Sep 05 00:02:10.195748 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:10.195714    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c922a1-85ce-484d-87cd-ef7b332acf56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7\" (UID: \"80c922a1-85ce-484d-87cd-ef7b332acf56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7"
Sep 05 00:02:10.196557 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:10.196530    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c922a1-85ce-484d-87cd-ef7b332acf56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7\" (UID: \"80c922a1-85ce-484d-87cd-ef7b332acf56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7"
Sep 05 00:02:10.311148 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:10.311069    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7"
Sep 05 00:02:10.582083 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:10.582016    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7"]
Sep 05 00:02:10.583199 ip-10-0-142-236 kubenswrapper[2557]: W0905 00:02:10.583174    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c922a1_85ce_484d_87cd_ef7b332acf56.slice/crio-db0629de21c3b701b10e87cdc6bcdec68d199f8d0bf5db09aff3cc16cd14106f WatchSource:0}: Error finding container db0629de21c3b701b10e87cdc6bcdec68d199f8d0bf5db09aff3cc16cd14106f: Status 404 returned error can't find the container with id db0629de21c3b701b10e87cdc6bcdec68d199f8d0bf5db09aff3cc16cd14106f
Sep 05 00:02:11.559318 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:11.559284    2557 generic.go:356] "Generic (PLEG): container finished" podID="80c922a1-85ce-484d-87cd-ef7b332acf56" containerID="0d966e1cf7e65ddc7e06af7e40431077733e69d80cd86d97fbbde3bf730c9d08" exitCode=0
Sep 05 00:02:11.559720 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:11.559372    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7" event={"ID":"80c922a1-85ce-484d-87cd-ef7b332acf56","Type":"ContainerDied","Data":"0d966e1cf7e65ddc7e06af7e40431077733e69d80cd86d97fbbde3bf730c9d08"}
Sep 05 00:02:11.559720 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:11.559421    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7" event={"ID":"80c922a1-85ce-484d-87cd-ef7b332acf56","Type":"ContainerStarted","Data":"db0629de21c3b701b10e87cdc6bcdec68d199f8d0bf5db09aff3cc16cd14106f"}
Sep 05 00:02:12.707635 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:12.707610    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7"
Sep 05 00:02:12.818990 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:12.818918    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c922a1-85ce-484d-87cd-ef7b332acf56-maas-api-service-ca\") pod \"80c922a1-85ce-484d-87cd-ef7b332acf56\" (UID: \"80c922a1-85ce-484d-87cd-ef7b332acf56\") "
Sep 05 00:02:12.819272 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:12.819250    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c922a1-85ce-484d-87cd-ef7b332acf56-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80c922a1-85ce-484d-87cd-ef7b332acf56" (UID: "80c922a1-85ce-484d-87cd-ef7b332acf56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:02:12.919991 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:12.919968    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c922a1-85ce-484d-87cd-ef7b332acf56-maas-api-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:13.572368 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:13.572341    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7"
Sep 05 00:02:13.572560 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:13.572388    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7" event={"ID":"80c922a1-85ce-484d-87cd-ef7b332acf56","Type":"ContainerDied","Data":"db0629de21c3b701b10e87cdc6bcdec68d199f8d0bf5db09aff3cc16cd14106f"}
Sep 05 00:02:13.572560 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:13.572445    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db0629de21c3b701b10e87cdc6bcdec68d199f8d0bf5db09aff3cc16cd14106f"
Sep 05 00:02:16.130732 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.130697    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"]
Sep 05 00:02:16.131298 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.130928    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h" podUID="ad411092-f763-4850-b6a9-e854493d2beb" containerName="payload-processing" containerID="cri-o://8d62627edef932d322d64c51454e4abf201a324cbcd30e91eadfe75a26271c7a" gracePeriod=30
Sep 05 00:02:16.280695 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.280662    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7"]
Sep 05 00:02:16.284875 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.284846    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3c355-k2qz7"]
Sep 05 00:02:16.401487 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.401461    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"
Sep 05 00:02:16.449287 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.449260    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad411092-f763-4850-b6a9-e854493d2beb-plugins-config-volume\") pod \"ad411092-f763-4850-b6a9-e854493d2beb\" (UID: \"ad411092-f763-4850-b6a9-e854493d2beb\") "
Sep 05 00:02:16.449462 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.449295    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr7w4\" (UniqueName: \"kubernetes.io/projected/ad411092-f763-4850-b6a9-e854493d2beb-kube-api-access-rr7w4\") pod \"ad411092-f763-4850-b6a9-e854493d2beb\" (UID: \"ad411092-f763-4850-b6a9-e854493d2beb\") "
Sep 05 00:02:16.449621 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.449601    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad411092-f763-4850-b6a9-e854493d2beb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad411092-f763-4850-b6a9-e854493d2beb" (UID: "ad411092-f763-4850-b6a9-e854493d2beb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:02:16.451431 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.451401    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad411092-f763-4850-b6a9-e854493d2beb-kube-api-access-rr7w4" (OuterVolumeSpecName: "kube-api-access-rr7w4") pod "ad411092-f763-4850-b6a9-e854493d2beb" (UID: "ad411092-f763-4850-b6a9-e854493d2beb"). InnerVolumeSpecName "kube-api-access-rr7w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:02:16.550122 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.550093    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr7w4\" (UniqueName: \"kubernetes.io/projected/ad411092-f763-4850-b6a9-e854493d2beb-kube-api-access-rr7w4\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:16.550122 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.550116    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad411092-f763-4850-b6a9-e854493d2beb-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:16.591667 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.591638    2557 generic.go:356] "Generic (PLEG): container finished" podID="ad411092-f763-4850-b6a9-e854493d2beb" containerID="8d62627edef932d322d64c51454e4abf201a324cbcd30e91eadfe75a26271c7a" exitCode=0
Sep 05 00:02:16.591798 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.591688    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"
Sep 05 00:02:16.591798 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.591730    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h" event={"ID":"ad411092-f763-4850-b6a9-e854493d2beb","Type":"ContainerDied","Data":"8d62627edef932d322d64c51454e4abf201a324cbcd30e91eadfe75a26271c7a"}
Sep 05 00:02:16.591798 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.591783    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h" event={"ID":"ad411092-f763-4850-b6a9-e854493d2beb","Type":"ContainerDied","Data":"2b2b6b4170522f191e0d17732f3b5eaeb3ad6316c9b4b8731e8138f20eba0bf2"}
Sep 05 00:02:16.591974 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.591809    2557 scope.go:117] "RemoveContainer" containerID="8d62627edef932d322d64c51454e4abf201a324cbcd30e91eadfe75a26271c7a"
Sep 05 00:02:16.601824 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.601805    2557 scope.go:117] "RemoveContainer" containerID="8d62627edef932d322d64c51454e4abf201a324cbcd30e91eadfe75a26271c7a"
Sep 05 00:02:16.602123 ip-10-0-142-236 kubenswrapper[2557]: E0905 00:02:16.602104    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d62627edef932d322d64c51454e4abf201a324cbcd30e91eadfe75a26271c7a\": container with ID starting with 8d62627edef932d322d64c51454e4abf201a324cbcd30e91eadfe75a26271c7a not found: ID does not exist" containerID="8d62627edef932d322d64c51454e4abf201a324cbcd30e91eadfe75a26271c7a"
Sep 05 00:02:16.602199 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.602136    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d62627edef932d322d64c51454e4abf201a324cbcd30e91eadfe75a26271c7a"} err="failed to get container status \"8d62627edef932d322d64c51454e4abf201a324cbcd30e91eadfe75a26271c7a\": rpc error: code = NotFound desc = could not find container \"8d62627edef932d322d64c51454e4abf201a324cbcd30e91eadfe75a26271c7a\": container with ID starting with 8d62627edef932d322d64c51454e4abf201a324cbcd30e91eadfe75a26271c7a not found: ID does not exist"
Sep 05 00:02:16.613822 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.613797    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"]
Sep 05 00:02:16.616211 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:16.616193    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b3c355-bd9f8978f-8fw6h"]
Sep 05 00:02:18.079856 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:18.079822    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80c922a1-85ce-484d-87cd-ef7b332acf56" path="/var/lib/kubelet/pods/80c922a1-85ce-484d-87cd-ef7b332acf56/volumes"
Sep 05 00:02:18.080366 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:18.080285    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad411092-f763-4850-b6a9-e854493d2beb" path="/var/lib/kubelet/pods/ad411092-f763-4850-b6a9-e854493d2beb/volumes"
Sep 05 00:02:19.713410 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:19.713376    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:02:22.472103 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:22.472028    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"]
Sep 05 00:02:22.472502 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:22.472279    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq" podUID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" containerName="istio-proxy" containerID="cri-o://0e021a26f3feed0059f4398a348b3268bd1bc6ad6df0b0fc139233b563e61051" gracePeriod=30
Sep 05 00:02:26.063418 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.063379    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2"]
Sep 05 00:02:26.063847 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.063833    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad411092-f763-4850-b6a9-e854493d2beb" containerName="payload-processing"
Sep 05 00:02:26.063893 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.063849    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad411092-f763-4850-b6a9-e854493d2beb" containerName="payload-processing"
Sep 05 00:02:26.063893 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.063863    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80c922a1-85ce-484d-87cd-ef7b332acf56" containerName="revoke-keys"
Sep 05 00:02:26.063893 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.063871    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="80c922a1-85ce-484d-87cd-ef7b332acf56" containerName="revoke-keys"
Sep 05 00:02:26.064033 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.063970    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="80c922a1-85ce-484d-87cd-ef7b332acf56" containerName="revoke-keys"
Sep 05 00:02:26.064033 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.063981    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad411092-f763-4850-b6a9-e854493d2beb" containerName="payload-processing"
Sep 05 00:02:26.067267 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.067249    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2"
Sep 05 00:02:26.069561 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.069540    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k4f64\""
Sep 05 00:02:26.069718 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.069688    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 00:02:26.081531 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.081506    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2"]
Sep 05 00:02:26.234691 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.234664    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec290395-41d1-4f05-861e-2d606910c5b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2\" (UID: \"ec290395-41d1-4f05-861e-2d606910c5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2"
Sep 05 00:02:26.336147 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.336066    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec290395-41d1-4f05-861e-2d606910c5b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2\" (UID: \"ec290395-41d1-4f05-861e-2d606910c5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2"
Sep 05 00:02:26.336718 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.336699    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec290395-41d1-4f05-861e-2d606910c5b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2\" (UID: \"ec290395-41d1-4f05-861e-2d606910c5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2"
Sep 05 00:02:26.380039 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.380014    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2"
Sep 05 00:02:26.651534 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:26.651506    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2"]
Sep 05 00:02:26.653268 ip-10-0-142-236 kubenswrapper[2557]: W0905 00:02:26.653242    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec290395_41d1_4f05_861e_2d606910c5b8.slice/crio-9786046d4de66a342343e7520ad8ed245a115b15f4eddb86778b00e6d2b2ae57 WatchSource:0}: Error finding container 9786046d4de66a342343e7520ad8ed245a115b15f4eddb86778b00e6d2b2ae57: Status 404 returned error can't find the container with id 9786046d4de66a342343e7520ad8ed245a115b15f4eddb86778b00e6d2b2ae57
Sep 05 00:02:27.656659 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.656626    2557 generic.go:356] "Generic (PLEG): container finished" podID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" containerID="0e021a26f3feed0059f4398a348b3268bd1bc6ad6df0b0fc139233b563e61051" exitCode=0
Sep 05 00:02:27.657055 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.656691    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq" event={"ID":"862597e0-ab54-4b16-b39b-bb1fe1b3811b","Type":"ContainerDied","Data":"0e021a26f3feed0059f4398a348b3268bd1bc6ad6df0b0fc139233b563e61051"}
Sep 05 00:02:27.657055 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.656750    2557 scope.go:117] "RemoveContainer" containerID="f5b0fef2593d6988a03726b0168ddaf33e7b3251e3b67cc69db0ef215f1f7d60"
Sep 05 00:02:27.658806 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.658783    2557 generic.go:356] "Generic (PLEG): container finished" podID="ec290395-41d1-4f05-861e-2d606910c5b8" containerID="5f1f98f7522c31908fa9a822fbabbba1d3dd9f1d89fbabb06e256a4d89e1c0ff" exitCode=0
Sep 05 00:02:27.658924 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.658846    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2" event={"ID":"ec290395-41d1-4f05-861e-2d606910c5b8","Type":"ContainerDied","Data":"5f1f98f7522c31908fa9a822fbabbba1d3dd9f1d89fbabb06e256a4d89e1c0ff"}
Sep 05 00:02:27.658924 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.658899    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2" event={"ID":"ec290395-41d1-4f05-861e-2d606910c5b8","Type":"ContainerStarted","Data":"9786046d4de66a342343e7520ad8ed245a115b15f4eddb86778b00e6d2b2ae57"}
Sep 05 00:02:27.728241 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.728220    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:02:27.849891 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.849823    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-credential-socket\") pod \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") "
Sep 05 00:02:27.849891 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.849858    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-podinfo\") pod \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") "
Sep 05 00:02:27.850114 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.849904    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54dw9\" (UniqueName: \"kubernetes.io/projected/862597e0-ab54-4b16-b39b-bb1fe1b3811b-kube-api-access-54dw9\") pod \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") "
Sep 05 00:02:27.850114 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.849967    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-workload-socket\") pod \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") "
Sep 05 00:02:27.850114 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.850097    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-workload-certs\") pod \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") "
Sep 05 00:02:27.850276 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.850156    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-data\") pod \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") "
Sep 05 00:02:27.850276 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.850188    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "862597e0-ab54-4b16-b39b-bb1fe1b3811b" (UID: "862597e0-ab54-4b16-b39b-bb1fe1b3811b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:02:27.850276 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.850203    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istiod-ca-cert\") pod \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") "
Sep 05 00:02:27.850276 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.850193    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "862597e0-ab54-4b16-b39b-bb1fe1b3811b" (UID: "862597e0-ab54-4b16-b39b-bb1fe1b3811b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:02:27.850276 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.850253    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-token\") pod \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") "
Sep 05 00:02:27.850543 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.850282    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-envoy\") pod \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\" (UID: \"862597e0-ab54-4b16-b39b-bb1fe1b3811b\") "
Sep 05 00:02:27.850543 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.850315    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "862597e0-ab54-4b16-b39b-bb1fe1b3811b" (UID: "862597e0-ab54-4b16-b39b-bb1fe1b3811b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:02:27.850543 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.850367    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-data" (OuterVolumeSpecName: "istio-data") pod "862597e0-ab54-4b16-b39b-bb1fe1b3811b" (UID: "862597e0-ab54-4b16-b39b-bb1fe1b3811b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:02:27.850704 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.850555    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "862597e0-ab54-4b16-b39b-bb1fe1b3811b" (UID: "862597e0-ab54-4b16-b39b-bb1fe1b3811b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:02:27.850704 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.850671    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:27.850704 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.850688    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:27.850704 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.850702    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:27.850916 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.850714    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:27.850916 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.850728    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:27.852250 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.852225    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "862597e0-ab54-4b16-b39b-bb1fe1b3811b" (UID: "862597e0-ab54-4b16-b39b-bb1fe1b3811b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 00:02:27.852250 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.852240    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-token" (OuterVolumeSpecName: "istio-token") pod "862597e0-ab54-4b16-b39b-bb1fe1b3811b" (UID: "862597e0-ab54-4b16-b39b-bb1fe1b3811b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:02:27.852392 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.852251    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862597e0-ab54-4b16-b39b-bb1fe1b3811b-kube-api-access-54dw9" (OuterVolumeSpecName: "kube-api-access-54dw9") pod "862597e0-ab54-4b16-b39b-bb1fe1b3811b" (UID: "862597e0-ab54-4b16-b39b-bb1fe1b3811b"). InnerVolumeSpecName "kube-api-access-54dw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:02:27.852392 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.852294    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "862597e0-ab54-4b16-b39b-bb1fe1b3811b" (UID: "862597e0-ab54-4b16-b39b-bb1fe1b3811b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:02:27.951594 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.951566    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:27.951594 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.951590    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:27.951594 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.951599    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/862597e0-ab54-4b16-b39b-bb1fe1b3811b-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:27.951807 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:27.951608    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54dw9\" (UniqueName: \"kubernetes.io/projected/862597e0-ab54-4b16-b39b-bb1fe1b3811b-kube-api-access-54dw9\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:28.666187 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:28.666155    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"
Sep 05 00:02:28.666609 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:28.666202    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq" event={"ID":"862597e0-ab54-4b16-b39b-bb1fe1b3811b","Type":"ContainerDied","Data":"dba532bc5a6d7ef4c7de395f6c2a163ecb019b1d2d01c0cb5b51fef76b97ae33"}
Sep 05 00:02:28.666609 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:28.666253    2557 scope.go:117] "RemoveContainer" containerID="0e021a26f3feed0059f4398a348b3268bd1bc6ad6df0b0fc139233b563e61051"
Sep 05 00:02:28.686985 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:28.686957    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"]
Sep 05 00:02:28.689718 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:28.689693    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b3c355-openshift-default-cdbf9b574-vcjxq"]
Sep 05 00:02:28.797683 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:28.797658    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2"
Sep 05 00:02:28.958643 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:28.958614    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec290395-41d1-4f05-861e-2d606910c5b8-maas-api-service-ca\") pod \"ec290395-41d1-4f05-861e-2d606910c5b8\" (UID: \"ec290395-41d1-4f05-861e-2d606910c5b8\") "
Sep 05 00:02:28.958960 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:28.958920    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec290395-41d1-4f05-861e-2d606910c5b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ec290395-41d1-4f05-861e-2d606910c5b8" (UID: "ec290395-41d1-4f05-861e-2d606910c5b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:02:29.059957 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:29.059901    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec290395-41d1-4f05-861e-2d606910c5b8-maas-api-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:29.675231 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:29.675197    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2" event={"ID":"ec290395-41d1-4f05-861e-2d606910c5b8","Type":"ContainerDied","Data":"9786046d4de66a342343e7520ad8ed245a115b15f4eddb86778b00e6d2b2ae57"}
Sep 05 00:02:29.675231 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:29.675215    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2"
Sep 05 00:02:29.675653 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:29.675266    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9786046d4de66a342343e7520ad8ed245a115b15f4eddb86778b00e6d2b2ae57"
Sep 05 00:02:30.079966 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:30.079919    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" path="/var/lib/kubelet/pods/862597e0-ab54-4b16-b39b-bb1fe1b3811b/volumes"
Sep 05 00:02:41.519223 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.519187    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"]
Sep 05 00:02:41.519933 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.519881    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr" podUID="095c5896-2de9-4c6d-8dfa-fcc215b3972d" containerName="payload-processing" containerID="cri-o://7e6c688ebe589533b967986aae1bc21241c4def6d5399008f2b1d63f188e076a" gracePeriod=30
Sep 05 00:02:41.529768 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.529742    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"]
Sep 05 00:02:41.530008 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.529983    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz" podUID="c22400c6-48cb-45d0-bbc3-2ed640e41a40" containerName="payload-pre-processing" containerID="cri-o://bfad216dc22a3d84242a3e5cf78abbb63214210b550a9eeb688169898173ab9f" gracePeriod=30
Sep 05 00:02:41.757918 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.757891    2557 generic.go:356] "Generic (PLEG): container finished" podID="095c5896-2de9-4c6d-8dfa-fcc215b3972d" containerID="7e6c688ebe589533b967986aae1bc21241c4def6d5399008f2b1d63f188e076a" exitCode=0
Sep 05 00:02:41.758083 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.757981    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr" event={"ID":"095c5896-2de9-4c6d-8dfa-fcc215b3972d","Type":"ContainerDied","Data":"7e6c688ebe589533b967986aae1bc21241c4def6d5399008f2b1d63f188e076a"}
Sep 05 00:02:41.760051 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.760030    2557 generic.go:356] "Generic (PLEG): container finished" podID="c22400c6-48cb-45d0-bbc3-2ed640e41a40" containerID="bfad216dc22a3d84242a3e5cf78abbb63214210b550a9eeb688169898173ab9f" exitCode=0
Sep 05 00:02:41.760202 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.760085    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz" event={"ID":"c22400c6-48cb-45d0-bbc3-2ed640e41a40","Type":"ContainerDied","Data":"bfad216dc22a3d84242a3e5cf78abbb63214210b550a9eeb688169898173ab9f"}
Sep 05 00:02:41.784154 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.783974    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"
Sep 05 00:02:41.785480 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.785459    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2"]
Sep 05 00:02:41.787859 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.787802    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea561-tdxd2"]
Sep 05 00:02:41.818280 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.818244    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"
Sep 05 00:02:41.960190 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.960156    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlfbl\" (UniqueName: \"kubernetes.io/projected/095c5896-2de9-4c6d-8dfa-fcc215b3972d-kube-api-access-dlfbl\") pod \"095c5896-2de9-4c6d-8dfa-fcc215b3972d\" (UID: \"095c5896-2de9-4c6d-8dfa-fcc215b3972d\") "
Sep 05 00:02:41.960333 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.960284    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22400c6-48cb-45d0-bbc3-2ed640e41a40-plugins-config-volume\") pod \"c22400c6-48cb-45d0-bbc3-2ed640e41a40\" (UID: \"c22400c6-48cb-45d0-bbc3-2ed640e41a40\") "
Sep 05 00:02:41.960333 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.960314    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mn6z8\" (UniqueName: \"kubernetes.io/projected/c22400c6-48cb-45d0-bbc3-2ed640e41a40-kube-api-access-mn6z8\") pod \"c22400c6-48cb-45d0-bbc3-2ed640e41a40\" (UID: \"c22400c6-48cb-45d0-bbc3-2ed640e41a40\") "
Sep 05 00:02:41.960428 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.960332    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095c5896-2de9-4c6d-8dfa-fcc215b3972d-plugins-config-volume\") pod \"095c5896-2de9-4c6d-8dfa-fcc215b3972d\" (UID: \"095c5896-2de9-4c6d-8dfa-fcc215b3972d\") "
Sep 05 00:02:41.960697 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.960661    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22400c6-48cb-45d0-bbc3-2ed640e41a40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c22400c6-48cb-45d0-bbc3-2ed640e41a40" (UID: "c22400c6-48cb-45d0-bbc3-2ed640e41a40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:02:41.960838 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.960732    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095c5896-2de9-4c6d-8dfa-fcc215b3972d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "095c5896-2de9-4c6d-8dfa-fcc215b3972d" (UID: "095c5896-2de9-4c6d-8dfa-fcc215b3972d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:02:41.962293 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.962266    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095c5896-2de9-4c6d-8dfa-fcc215b3972d-kube-api-access-dlfbl" (OuterVolumeSpecName: "kube-api-access-dlfbl") pod "095c5896-2de9-4c6d-8dfa-fcc215b3972d" (UID: "095c5896-2de9-4c6d-8dfa-fcc215b3972d"). InnerVolumeSpecName "kube-api-access-dlfbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:02:41.962376 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:41.962310    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22400c6-48cb-45d0-bbc3-2ed640e41a40-kube-api-access-mn6z8" (OuterVolumeSpecName: "kube-api-access-mn6z8") pod "c22400c6-48cb-45d0-bbc3-2ed640e41a40" (UID: "c22400c6-48cb-45d0-bbc3-2ed640e41a40"). InnerVolumeSpecName "kube-api-access-mn6z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:02:42.061686 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.061632    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn6z8\" (UniqueName: \"kubernetes.io/projected/c22400c6-48cb-45d0-bbc3-2ed640e41a40-kube-api-access-mn6z8\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:42.061686 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.061655    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095c5896-2de9-4c6d-8dfa-fcc215b3972d-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:42.061686 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.061665    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlfbl\" (UniqueName: \"kubernetes.io/projected/095c5896-2de9-4c6d-8dfa-fcc215b3972d-kube-api-access-dlfbl\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:42.061686 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.061674    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22400c6-48cb-45d0-bbc3-2ed640e41a40-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:02:42.078602 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.078576    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec290395-41d1-4f05-861e-2d606910c5b8" path="/var/lib/kubelet/pods/ec290395-41d1-4f05-861e-2d606910c5b8/volumes"
Sep 05 00:02:42.220446 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.220420    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:02:42.766692 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.766661    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"
Sep 05 00:02:42.767122 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.766670    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr" event={"ID":"095c5896-2de9-4c6d-8dfa-fcc215b3972d","Type":"ContainerDied","Data":"7df91d5065552014089ed8446bdb44054782dafff5c1691b098206d15ba1e363"}
Sep 05 00:02:42.767122 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.766800    2557 scope.go:117] "RemoveContainer" containerID="7e6c688ebe589533b967986aae1bc21241c4def6d5399008f2b1d63f188e076a"
Sep 05 00:02:42.768902 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.768877    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"
Sep 05 00:02:42.769051 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.768901    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz" event={"ID":"c22400c6-48cb-45d0-bbc3-2ed640e41a40","Type":"ContainerDied","Data":"8d7794be5a5711e7102f02b6b814423da284399481b988951fd916d3d1c4545b"}
Sep 05 00:02:42.778172 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.778150    2557 scope.go:117] "RemoveContainer" containerID="bfad216dc22a3d84242a3e5cf78abbb63214210b550a9eeb688169898173ab9f"
Sep 05 00:02:42.786436 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.786415    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"]
Sep 05 00:02:42.790207 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.790186    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bea561-5cf94c685d-5hdxz"]
Sep 05 00:02:42.800221 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.800202    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"]
Sep 05 00:02:42.802337 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:42.802307    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bea561-d95ffd8cc-kblhr"]
Sep 05 00:02:44.079245 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:44.079216    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="095c5896-2de9-4c6d-8dfa-fcc215b3972d" path="/var/lib/kubelet/pods/095c5896-2de9-4c6d-8dfa-fcc215b3972d/volumes"
Sep 05 00:02:44.079618 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:44.079559    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c22400c6-48cb-45d0-bbc3-2ed640e41a40" path="/var/lib/kubelet/pods/c22400c6-48cb-45d0-bbc3-2ed640e41a40/volumes"
Sep 05 00:02:47.140769 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:47.140737    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"]
Sep 05 00:02:52.212877 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:52.212848    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:02:58.386605 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:02:58.386575    2557 scope.go:117] "RemoveContainer" containerID="2599cf4843a96684e83d69bcf19cd9965024b1c2190a3f61c1ba72a11362c280"
Sep 05 00:03:02.160561 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.160520    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq" podUID="9965b46e-1c91-4f94-b7fe-1ae9303659d9" containerName="main" containerID="cri-o://069fcb4a467326080bf59a746193dd14e18cf6fd5342cd8c2084b7257ad9288a" gracePeriod=45
Sep 05 00:03:02.401737 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.401711    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:03:02.520887 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.520857    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9965b46e-1c91-4f94-b7fe-1ae9303659d9-tls-certs\") pod \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") "
Sep 05 00:03:02.521105 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.520913    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-dshm\") pod \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") "
Sep 05 00:03:02.521105 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.520959    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-home\") pod \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") "
Sep 05 00:03:02.521105 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.521067    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jl99\" (UniqueName: \"kubernetes.io/projected/9965b46e-1c91-4f94-b7fe-1ae9303659d9-kube-api-access-2jl99\") pod \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") "
Sep 05 00:03:02.521283 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.521115    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-tmp-dir\") pod \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") "
Sep 05 00:03:02.521283 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.521142    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-model-cache\") pod \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\" (UID: \"9965b46e-1c91-4f94-b7fe-1ae9303659d9\") "
Sep 05 00:03:02.521283 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.521187    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-home" (OuterVolumeSpecName: "home") pod "9965b46e-1c91-4f94-b7fe-1ae9303659d9" (UID: "9965b46e-1c91-4f94-b7fe-1ae9303659d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:03:02.521447 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.521382    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9965b46e-1c91-4f94-b7fe-1ae9303659d9" (UID: "9965b46e-1c91-4f94-b7fe-1ae9303659d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:03:02.521506 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.521461    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:02.521506 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.521482    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:02.521506 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.521473    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "9965b46e-1c91-4f94-b7fe-1ae9303659d9" (UID: "9965b46e-1c91-4f94-b7fe-1ae9303659d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:03:02.523297 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.523267    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9965b46e-1c91-4f94-b7fe-1ae9303659d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9965b46e-1c91-4f94-b7fe-1ae9303659d9" (UID: "9965b46e-1c91-4f94-b7fe-1ae9303659d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:03:02.523428 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.523350    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-dshm" (OuterVolumeSpecName: "dshm") pod "9965b46e-1c91-4f94-b7fe-1ae9303659d9" (UID: "9965b46e-1c91-4f94-b7fe-1ae9303659d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:03:02.523428 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.523357    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9965b46e-1c91-4f94-b7fe-1ae9303659d9-kube-api-access-2jl99" (OuterVolumeSpecName: "kube-api-access-2jl99") pod "9965b46e-1c91-4f94-b7fe-1ae9303659d9" (UID: "9965b46e-1c91-4f94-b7fe-1ae9303659d9"). InnerVolumeSpecName "kube-api-access-2jl99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:03:02.622745 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.622713    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9965b46e-1c91-4f94-b7fe-1ae9303659d9-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:02.622745 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.622740    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:02.622903 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.622753    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jl99\" (UniqueName: \"kubernetes.io/projected/9965b46e-1c91-4f94-b7fe-1ae9303659d9-kube-api-access-2jl99\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:02.622903 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.622766    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9965b46e-1c91-4f94-b7fe-1ae9303659d9-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:02.890344 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.890266    2557 generic.go:356] "Generic (PLEG): container finished" podID="9965b46e-1c91-4f94-b7fe-1ae9303659d9" containerID="069fcb4a467326080bf59a746193dd14e18cf6fd5342cd8c2084b7257ad9288a" exitCode=0
Sep 05 00:03:02.890344 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.890310    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"
Sep 05 00:03:02.890507 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.890352    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq" event={"ID":"9965b46e-1c91-4f94-b7fe-1ae9303659d9","Type":"ContainerDied","Data":"069fcb4a467326080bf59a746193dd14e18cf6fd5342cd8c2084b7257ad9288a"}
Sep 05 00:03:02.890507 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.890399    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq" event={"ID":"9965b46e-1c91-4f94-b7fe-1ae9303659d9","Type":"ContainerDied","Data":"90b5255c4d58e9e316afb84db04787df40de48d38131f8b8b5625fbb569dc0eb"}
Sep 05 00:03:02.890507 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.890422    2557 scope.go:117] "RemoveContainer" containerID="069fcb4a467326080bf59a746193dd14e18cf6fd5342cd8c2084b7257ad9288a"
Sep 05 00:03:02.899698 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.899680    2557 scope.go:117] "RemoveContainer" containerID="069fcb4a467326080bf59a746193dd14e18cf6fd5342cd8c2084b7257ad9288a"
Sep 05 00:03:02.899978 ip-10-0-142-236 kubenswrapper[2557]: E0905 00:03:02.899959    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"069fcb4a467326080bf59a746193dd14e18cf6fd5342cd8c2084b7257ad9288a\": container with ID starting with 069fcb4a467326080bf59a746193dd14e18cf6fd5342cd8c2084b7257ad9288a not found: ID does not exist" containerID="069fcb4a467326080bf59a746193dd14e18cf6fd5342cd8c2084b7257ad9288a"
Sep 05 00:03:02.900058 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.899986    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"069fcb4a467326080bf59a746193dd14e18cf6fd5342cd8c2084b7257ad9288a"} err="failed to get container status \"069fcb4a467326080bf59a746193dd14e18cf6fd5342cd8c2084b7257ad9288a\": rpc error: code = NotFound desc = could not find container \"069fcb4a467326080bf59a746193dd14e18cf6fd5342cd8c2084b7257ad9288a\": container with ID starting with 069fcb4a467326080bf59a746193dd14e18cf6fd5342cd8c2084b7257ad9288a not found: ID does not exist"
Sep 05 00:03:02.911984 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.911964    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"]
Sep 05 00:03:02.915198 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:02.915178    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-bea561/ipp-route-bea561-kserve-6fb7f58787-58zrq"]
Sep 05 00:03:04.079475 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:04.079438    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9965b46e-1c91-4f94-b7fe-1ae9303659d9" path="/var/lib/kubelet/pods/9965b46e-1c91-4f94-b7fe-1ae9303659d9/volumes"
Sep 05 00:03:30.344372 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344336    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st"]
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344689    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="095c5896-2de9-4c6d-8dfa-fcc215b3972d" containerName="payload-processing"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344700    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="095c5896-2de9-4c6d-8dfa-fcc215b3972d" containerName="payload-processing"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344713    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22400c6-48cb-45d0-bbc3-2ed640e41a40" containerName="payload-pre-processing"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344718    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22400c6-48cb-45d0-bbc3-2ed640e41a40" containerName="payload-pre-processing"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344727    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9965b46e-1c91-4f94-b7fe-1ae9303659d9" containerName="main"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344733    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9965b46e-1c91-4f94-b7fe-1ae9303659d9" containerName="main"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344746    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec290395-41d1-4f05-861e-2d606910c5b8" containerName="revoke-keys"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344751    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec290395-41d1-4f05-861e-2d606910c5b8" containerName="revoke-keys"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344761    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" containerName="istio-proxy"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344766    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" containerName="istio-proxy"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344771    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" containerName="istio-proxy"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344776    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" containerName="istio-proxy"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344833    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22400c6-48cb-45d0-bbc3-2ed640e41a40" containerName="payload-pre-processing"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344843    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" containerName="istio-proxy"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344849    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="862597e0-ab54-4b16-b39b-bb1fe1b3811b" containerName="istio-proxy"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344854    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec290395-41d1-4f05-861e-2d606910c5b8" containerName="revoke-keys"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344862    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="095c5896-2de9-4c6d-8dfa-fcc215b3972d" containerName="payload-processing"
Sep 05 00:03:30.344883 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.344867    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9965b46e-1c91-4f94-b7fe-1ae9303659d9" containerName="main"
Sep 05 00:03:30.346745 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.346730    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st"
Sep 05 00:03:30.349437 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.349414    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k4f64\""
Sep 05 00:03:30.349568 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.349441    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 00:03:30.354631 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.354472    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st"]
Sep 05 00:03:30.461810 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.461769    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18251201-ffca-4c60-8078-df6db6b72685-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st\" (UID: \"18251201-ffca-4c60-8078-df6db6b72685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st"
Sep 05 00:03:30.562709 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.562596    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18251201-ffca-4c60-8078-df6db6b72685-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st\" (UID: \"18251201-ffca-4c60-8078-df6db6b72685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st"
Sep 05 00:03:30.563298 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.563277    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18251201-ffca-4c60-8078-df6db6b72685-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st\" (UID: \"18251201-ffca-4c60-8078-df6db6b72685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st"
Sep 05 00:03:30.658702 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.658641    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st"
Sep 05 00:03:30.941825 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:30.941797    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st"]
Sep 05 00:03:30.944092 ip-10-0-142-236 kubenswrapper[2557]: W0905 00:03:30.944064    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18251201_ffca_4c60_8078_df6db6b72685.slice/crio-96714e5e6f68b47e546e3f0d7dae645b9fbbfe45de99b75f284e49bc472eed9d WatchSource:0}: Error finding container 96714e5e6f68b47e546e3f0d7dae645b9fbbfe45de99b75f284e49bc472eed9d: Status 404 returned error can't find the container with id 96714e5e6f68b47e546e3f0d7dae645b9fbbfe45de99b75f284e49bc472eed9d
Sep 05 00:03:31.050365 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:31.050337    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st" event={"ID":"18251201-ffca-4c60-8078-df6db6b72685","Type":"ContainerStarted","Data":"96714e5e6f68b47e546e3f0d7dae645b9fbbfe45de99b75f284e49bc472eed9d"}
Sep 05 00:03:32.058897 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:32.058865    2557 generic.go:356] "Generic (PLEG): container finished" podID="18251201-ffca-4c60-8078-df6db6b72685" containerID="d7ecd50512e453102d78663a7c1324b47ec9de0d1e4b2468e33d1c436c3c82c0" exitCode=0
Sep 05 00:03:32.059299 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:32.058901    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st" event={"ID":"18251201-ffca-4c60-8078-df6db6b72685","Type":"ContainerDied","Data":"d7ecd50512e453102d78663a7c1324b47ec9de0d1e4b2468e33d1c436c3c82c0"}
Sep 05 00:03:33.199046 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:33.199022    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st"
Sep 05 00:03:33.388633 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:33.388570    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18251201-ffca-4c60-8078-df6db6b72685-maas-api-service-ca\") pod \"18251201-ffca-4c60-8078-df6db6b72685\" (UID: \"18251201-ffca-4c60-8078-df6db6b72685\") "
Sep 05 00:03:33.388886 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:33.388866    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18251201-ffca-4c60-8078-df6db6b72685-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18251201-ffca-4c60-8078-df6db6b72685" (UID: "18251201-ffca-4c60-8078-df6db6b72685"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:03:33.489969 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:33.489930    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18251201-ffca-4c60-8078-df6db6b72685-maas-api-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:34.071222 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:34.071196    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st"
Sep 05 00:03:34.071379 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:34.071243    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st" event={"ID":"18251201-ffca-4c60-8078-df6db6b72685","Type":"ContainerDied","Data":"96714e5e6f68b47e546e3f0d7dae645b9fbbfe45de99b75f284e49bc472eed9d"}
Sep 05 00:03:34.071379 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:34.071293    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96714e5e6f68b47e546e3f0d7dae645b9fbbfe45de99b75f284e49bc472eed9d"
Sep 05 00:03:40.968788 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:40.968751    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st"]
Sep 05 00:03:40.971309 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:40.971281    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64702c-wl2st"]
Sep 05 00:03:42.078462 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:42.078431    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18251201-ffca-4c60-8078-df6db6b72685" path="/var/lib/kubelet/pods/18251201-ffca-4c60-8078-df6db6b72685/volumes"
Sep 05 00:03:44.910067 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:44.910034    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:03:47.180926 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:47.180896    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"]
Sep 05 00:03:47.181344 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:47.181150    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv" podUID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" containerName="istio-proxy" containerID="cri-o://101d9e9b27416c648c16c106d1eb0cbd39a3934d3c5ef24e8d27b98ac83246d0" gracePeriod=30
Sep 05 00:03:48.962879 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:48.962844    2557 patch_prober.go:28] interesting pod/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 05 00:03:48.963279 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:48.962908    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv" podUID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 00:03:52.435111 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.435090    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:03:52.539560 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.539536    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-workload-certs\") pod \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") "
Sep 05 00:03:52.539680 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.539570    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2b2d\" (UniqueName: \"kubernetes.io/projected/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-kube-api-access-h2b2d\") pod \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") "
Sep 05 00:03:52.539680 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.539597    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-token\") pod \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") "
Sep 05 00:03:52.539680 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.539625    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-envoy\") pod \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") "
Sep 05 00:03:52.539808 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.539730    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istiod-ca-cert\") pod \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") "
Sep 05 00:03:52.539808 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.539765    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-workload-socket\") pod \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") "
Sep 05 00:03:52.539922 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.539810    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-podinfo\") pod \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") "
Sep 05 00:03:52.539922 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.539836    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-credential-socket\") pod \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") "
Sep 05 00:03:52.539922 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.539872    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-data\") pod \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\" (UID: \"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07\") "
Sep 05 00:03:52.539922 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.539881    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" (UID: "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:03:52.540166 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.540084    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" (UID: "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:03:52.540166 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.540128    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" (UID: "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:03:52.540317 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.540274    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" (UID: "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:03:52.540416 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.540390    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:52.540484 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.540398    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" (UID: "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:03:52.540484 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.540421    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:52.540484 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.540471    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:52.540595 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.540484    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:52.541756 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.541738    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-kube-api-access-h2b2d" (OuterVolumeSpecName: "kube-api-access-h2b2d") pod "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" (UID: "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07"). InnerVolumeSpecName "kube-api-access-h2b2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:03:52.541905 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.541885    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" (UID: "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 00:03:52.542201 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.542174    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" (UID: "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:03:52.542201 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.542184    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" (UID: "3c367d4c-60b9-4fb2-afbe-3fc458fb3a07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:03:52.641865 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.641845    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:52.641865 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.641864    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:52.642007 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.641873    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:52.642007 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.641883    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2b2d\" (UniqueName: \"kubernetes.io/projected/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-kube-api-access-h2b2d\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:52.642007 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:52.641892    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:53.179073 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:53.179047    2557 generic.go:356] "Generic (PLEG): container finished" podID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" containerID="101d9e9b27416c648c16c106d1eb0cbd39a3934d3c5ef24e8d27b98ac83246d0" exitCode=0
Sep 05 00:03:53.179233 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:53.179088    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"
Sep 05 00:03:53.179233 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:53.179130    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv" event={"ID":"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07","Type":"ContainerDied","Data":"101d9e9b27416c648c16c106d1eb0cbd39a3934d3c5ef24e8d27b98ac83246d0"}
Sep 05 00:03:53.179233 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:53.179178    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv" event={"ID":"3c367d4c-60b9-4fb2-afbe-3fc458fb3a07","Type":"ContainerDied","Data":"116d31de2b9e81e6ed48be2ecf2d5218c2c1aad2de7517e84f5ff601d34ae768"}
Sep 05 00:03:53.179233 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:53.179199    2557 scope.go:117] "RemoveContainer" containerID="101d9e9b27416c648c16c106d1eb0cbd39a3934d3c5ef24e8d27b98ac83246d0"
Sep 05 00:03:53.189107 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:53.189087    2557 scope.go:117] "RemoveContainer" containerID="f1ecc175e669688a17e892e4757aac4db3a3c6487c2e52d33f3948321abaddc5"
Sep 05 00:03:53.197186 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:53.197169    2557 scope.go:117] "RemoveContainer" containerID="101d9e9b27416c648c16c106d1eb0cbd39a3934d3c5ef24e8d27b98ac83246d0"
Sep 05 00:03:53.197460 ip-10-0-142-236 kubenswrapper[2557]: E0905 00:03:53.197442    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101d9e9b27416c648c16c106d1eb0cbd39a3934d3c5ef24e8d27b98ac83246d0\": container with ID starting with 101d9e9b27416c648c16c106d1eb0cbd39a3934d3c5ef24e8d27b98ac83246d0 not found: ID does not exist" containerID="101d9e9b27416c648c16c106d1eb0cbd39a3934d3c5ef24e8d27b98ac83246d0"
Sep 05 00:03:53.197536 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:53.197473    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101d9e9b27416c648c16c106d1eb0cbd39a3934d3c5ef24e8d27b98ac83246d0"} err="failed to get container status \"101d9e9b27416c648c16c106d1eb0cbd39a3934d3c5ef24e8d27b98ac83246d0\": rpc error: code = NotFound desc = could not find container \"101d9e9b27416c648c16c106d1eb0cbd39a3934d3c5ef24e8d27b98ac83246d0\": container with ID starting with 101d9e9b27416c648c16c106d1eb0cbd39a3934d3c5ef24e8d27b98ac83246d0 not found: ID does not exist"
Sep 05 00:03:53.197536 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:53.197495    2557 scope.go:117] "RemoveContainer" containerID="f1ecc175e669688a17e892e4757aac4db3a3c6487c2e52d33f3948321abaddc5"
Sep 05 00:03:53.197797 ip-10-0-142-236 kubenswrapper[2557]: E0905 00:03:53.197777    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ecc175e669688a17e892e4757aac4db3a3c6487c2e52d33f3948321abaddc5\": container with ID starting with f1ecc175e669688a17e892e4757aac4db3a3c6487c2e52d33f3948321abaddc5 not found: ID does not exist" containerID="f1ecc175e669688a17e892e4757aac4db3a3c6487c2e52d33f3948321abaddc5"
Sep 05 00:03:53.197839 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:53.197804    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ecc175e669688a17e892e4757aac4db3a3c6487c2e52d33f3948321abaddc5"} err="failed to get container status \"f1ecc175e669688a17e892e4757aac4db3a3c6487c2e52d33f3948321abaddc5\": rpc error: code = NotFound desc = could not find container \"f1ecc175e669688a17e892e4757aac4db3a3c6487c2e52d33f3948321abaddc5\": container with ID starting with f1ecc175e669688a17e892e4757aac4db3a3c6487c2e52d33f3948321abaddc5 not found: ID does not exist"
Sep 05 00:03:53.201781 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:53.201759    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"]
Sep 05 00:03:53.205175 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:53.205147    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-64702c-openshift-default-5fc78b848d-vhhsv"]
Sep 05 00:03:54.080224 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:54.080190    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" path="/var/lib/kubelet/pods/3c367d4c-60b9-4fb2-afbe-3fc458fb3a07/volumes"
Sep 05 00:03:58.013227 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:58.013195    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"]
Sep 05 00:03:58.013662 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:58.013400    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6" podUID="c1ded87b-823c-46c2-be19-513efbcc60b3" containerName="payload-pre-processing" containerID="cri-o://a3204f0e6513b2010f7438fafa6c12ccbc8bfe75d338d6ceeacb70815fb2f449" gracePeriod=30
Sep 05 00:03:58.215923 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:58.215891    2557 generic.go:356] "Generic (PLEG): container finished" podID="c1ded87b-823c-46c2-be19-513efbcc60b3" containerID="a3204f0e6513b2010f7438fafa6c12ccbc8bfe75d338d6ceeacb70815fb2f449" exitCode=0
Sep 05 00:03:58.216071 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:58.215966    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6" event={"ID":"c1ded87b-823c-46c2-be19-513efbcc60b3","Type":"ContainerDied","Data":"a3204f0e6513b2010f7438fafa6c12ccbc8bfe75d338d6ceeacb70815fb2f449"}
Sep 05 00:03:58.262830 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:58.262809    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"
Sep 05 00:03:58.286668 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:58.286601    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ded87b-823c-46c2-be19-513efbcc60b3-plugins-config-volume\") pod \"c1ded87b-823c-46c2-be19-513efbcc60b3\" (UID: \"c1ded87b-823c-46c2-be19-513efbcc60b3\") "
Sep 05 00:03:58.286804 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:58.286670    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jsg9\" (UniqueName: \"kubernetes.io/projected/c1ded87b-823c-46c2-be19-513efbcc60b3-kube-api-access-9jsg9\") pod \"c1ded87b-823c-46c2-be19-513efbcc60b3\" (UID: \"c1ded87b-823c-46c2-be19-513efbcc60b3\") "
Sep 05 00:03:58.287067 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:58.287037    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ded87b-823c-46c2-be19-513efbcc60b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1ded87b-823c-46c2-be19-513efbcc60b3" (UID: "c1ded87b-823c-46c2-be19-513efbcc60b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:03:58.288830 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:58.288804    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ded87b-823c-46c2-be19-513efbcc60b3-kube-api-access-9jsg9" (OuterVolumeSpecName: "kube-api-access-9jsg9") pod "c1ded87b-823c-46c2-be19-513efbcc60b3" (UID: "c1ded87b-823c-46c2-be19-513efbcc60b3"). InnerVolumeSpecName "kube-api-access-9jsg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:03:58.388026 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:58.387991    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ded87b-823c-46c2-be19-513efbcc60b3-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:58.388026 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:58.388021    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jsg9\" (UniqueName: \"kubernetes.io/projected/c1ded87b-823c-46c2-be19-513efbcc60b3-kube-api-access-9jsg9\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:03:58.424235 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:58.424207    2557 scope.go:117] "RemoveContainer" containerID="a3204f0e6513b2010f7438fafa6c12ccbc8bfe75d338d6ceeacb70815fb2f449"
Sep 05 00:03:59.221048 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:59.221006    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6" event={"ID":"c1ded87b-823c-46c2-be19-513efbcc60b3","Type":"ContainerDied","Data":"71c2aaa862a6c8664de28e443e1627312f9916aaabd1acd7cd8245a538b014f7"}
Sep 05 00:03:59.221539 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:59.221099    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"
Sep 05 00:03:59.250664 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:59.250636    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"]
Sep 05 00:03:59.254532 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:03:59.254510    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5245c1-8f77bfb98-s95n6"]
Sep 05 00:04:00.078620 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:00.078586    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1ded87b-823c-46c2-be19-513efbcc60b3" path="/var/lib/kubelet/pods/c1ded87b-823c-46c2-be19-513efbcc60b3/volumes"
Sep 05 00:04:01.213170 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:01.213140    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:04:04.824685 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.824652    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s"]
Sep 05 00:04:04.825077 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.825002    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" containerName="istio-proxy"
Sep 05 00:04:04.825077 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.825015    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" containerName="istio-proxy"
Sep 05 00:04:04.825077 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.825025    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1ded87b-823c-46c2-be19-513efbcc60b3" containerName="payload-pre-processing"
Sep 05 00:04:04.825077 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.825030    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1ded87b-823c-46c2-be19-513efbcc60b3" containerName="payload-pre-processing"
Sep 05 00:04:04.825077 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.825037    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18251201-ffca-4c60-8078-df6db6b72685" containerName="revoke-keys"
Sep 05 00:04:04.825077 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.825043    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="18251201-ffca-4c60-8078-df6db6b72685" containerName="revoke-keys"
Sep 05 00:04:04.825077 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.825057    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" containerName="istio-proxy"
Sep 05 00:04:04.825077 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.825062    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" containerName="istio-proxy"
Sep 05 00:04:04.825339 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.825114    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" containerName="istio-proxy"
Sep 05 00:04:04.825339 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.825122    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c367d4c-60b9-4fb2-afbe-3fc458fb3a07" containerName="istio-proxy"
Sep 05 00:04:04.825339 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.825128    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="18251201-ffca-4c60-8078-df6db6b72685" containerName="revoke-keys"
Sep 05 00:04:04.825339 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.825134    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1ded87b-823c-46c2-be19-513efbcc60b3" containerName="payload-pre-processing"
Sep 05 00:04:04.828239 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.828224    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s"
Sep 05 00:04:04.830792 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.830742    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k4f64\""
Sep 05 00:04:04.830970 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.830864    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 00:04:04.834499 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.834479    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s"]
Sep 05 00:04:04.939347 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:04.939316    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb172db-b007-4250-a82f-6a2603d70d68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s\" (UID: \"0cb172db-b007-4250-a82f-6a2603d70d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s"
Sep 05 00:04:05.039718 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:05.039685    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb172db-b007-4250-a82f-6a2603d70d68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s\" (UID: \"0cb172db-b007-4250-a82f-6a2603d70d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s"
Sep 05 00:04:05.040393 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:05.040369    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb172db-b007-4250-a82f-6a2603d70d68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s\" (UID: \"0cb172db-b007-4250-a82f-6a2603d70d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s"
Sep 05 00:04:05.140742 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:05.140666    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s"
Sep 05 00:04:05.410329 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:05.410298    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s"]
Sep 05 00:04:05.412219 ip-10-0-142-236 kubenswrapper[2557]: W0905 00:04:05.412192    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb172db_b007_4250_a82f_6a2603d70d68.slice/crio-71d6cd28b2dac84ab5aa2054003db989fa95ce2a8fb9d06b9cf969c4dc6e1293 WatchSource:0}: Error finding container 71d6cd28b2dac84ab5aa2054003db989fa95ce2a8fb9d06b9cf969c4dc6e1293: Status 404 returned error can't find the container with id 71d6cd28b2dac84ab5aa2054003db989fa95ce2a8fb9d06b9cf969c4dc6e1293
Sep 05 00:04:06.259562 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:06.259489    2557 generic.go:356] "Generic (PLEG): container finished" podID="0cb172db-b007-4250-a82f-6a2603d70d68" containerID="5eabde5b40638a52519aef1d8029535bf981eb8c6f91e1398c2fffab0521b91b" exitCode=0
Sep 05 00:04:06.259885 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:06.259570    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s" event={"ID":"0cb172db-b007-4250-a82f-6a2603d70d68","Type":"ContainerDied","Data":"5eabde5b40638a52519aef1d8029535bf981eb8c6f91e1398c2fffab0521b91b"}
Sep 05 00:04:06.259885 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:06.259616    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s" event={"ID":"0cb172db-b007-4250-a82f-6a2603d70d68","Type":"ContainerStarted","Data":"71d6cd28b2dac84ab5aa2054003db989fa95ce2a8fb9d06b9cf969c4dc6e1293"}
Sep 05 00:04:07.394588 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:07.394565    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s"
Sep 05 00:04:07.459707 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:07.459669    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb172db-b007-4250-a82f-6a2603d70d68-maas-api-service-ca\") pod \"0cb172db-b007-4250-a82f-6a2603d70d68\" (UID: \"0cb172db-b007-4250-a82f-6a2603d70d68\") "
Sep 05 00:04:07.460021 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:07.459999    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb172db-b007-4250-a82f-6a2603d70d68-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0cb172db-b007-4250-a82f-6a2603d70d68" (UID: "0cb172db-b007-4250-a82f-6a2603d70d68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:04:07.561112 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:07.561057    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb172db-b007-4250-a82f-6a2603d70d68-maas-api-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:04:08.271583 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:08.271555    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s"
Sep 05 00:04:08.271756 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:08.271586    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s" event={"ID":"0cb172db-b007-4250-a82f-6a2603d70d68","Type":"ContainerDied","Data":"71d6cd28b2dac84ab5aa2054003db989fa95ce2a8fb9d06b9cf969c4dc6e1293"}
Sep 05 00:04:08.271756 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:08.271634    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71d6cd28b2dac84ab5aa2054003db989fa95ce2a8fb9d06b9cf969c4dc6e1293"
Sep 05 00:04:15.468177 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:15.468131    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-88828d-29809440-mhcpl"]
Sep 05 00:04:15.472185 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:15.472151    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-88828d-29809440-mhcpl"]
Sep 05 00:04:15.508432 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:15.508394    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"]
Sep 05 00:04:15.508749 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:15.508718    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt" podUID="d8bdac91-7d4e-48a9-a384-1eaf25625fd3" containerName="payload-pre-processing" containerID="cri-o://7c81445d524f2ca60760ebcc30c1530477d78378b24d2253e2c6d2431324f48b" gracePeriod=30
Sep 05 00:04:15.655355 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:15.655317    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s"]
Sep 05 00:04:15.657625 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:15.657604    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-88828d-l622s"]
Sep 05 00:04:15.760436 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:15.760407    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"
Sep 05 00:04:15.823852 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:15.823805    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8bdac91-7d4e-48a9-a384-1eaf25625fd3-plugins-config-volume\") pod \"d8bdac91-7d4e-48a9-a384-1eaf25625fd3\" (UID: \"d8bdac91-7d4e-48a9-a384-1eaf25625fd3\") "
Sep 05 00:04:15.823852 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:15.823866    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jg84\" (UniqueName: \"kubernetes.io/projected/d8bdac91-7d4e-48a9-a384-1eaf25625fd3-kube-api-access-4jg84\") pod \"d8bdac91-7d4e-48a9-a384-1eaf25625fd3\" (UID: \"d8bdac91-7d4e-48a9-a384-1eaf25625fd3\") "
Sep 05 00:04:15.824257 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:15.824228    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8bdac91-7d4e-48a9-a384-1eaf25625fd3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8bdac91-7d4e-48a9-a384-1eaf25625fd3" (UID: "d8bdac91-7d4e-48a9-a384-1eaf25625fd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:04:15.825986 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:15.825924    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8bdac91-7d4e-48a9-a384-1eaf25625fd3-kube-api-access-4jg84" (OuterVolumeSpecName: "kube-api-access-4jg84") pod "d8bdac91-7d4e-48a9-a384-1eaf25625fd3" (UID: "d8bdac91-7d4e-48a9-a384-1eaf25625fd3"). InnerVolumeSpecName "kube-api-access-4jg84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:04:15.925049 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:15.925016    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jg84\" (UniqueName: \"kubernetes.io/projected/d8bdac91-7d4e-48a9-a384-1eaf25625fd3-kube-api-access-4jg84\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:04:15.925049 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:15.925045    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8bdac91-7d4e-48a9-a384-1eaf25625fd3-plugins-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:04:16.079147 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:16.079073    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cb172db-b007-4250-a82f-6a2603d70d68" path="/var/lib/kubelet/pods/0cb172db-b007-4250-a82f-6a2603d70d68/volumes"
Sep 05 00:04:16.079373 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:16.079360    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="619f348d-1041-4016-b884-c1ed013c8a97" path="/var/lib/kubelet/pods/619f348d-1041-4016-b884-c1ed013c8a97/volumes"
Sep 05 00:04:16.314468 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:16.314434    2557 generic.go:356] "Generic (PLEG): container finished" podID="d8bdac91-7d4e-48a9-a384-1eaf25625fd3" containerID="7c81445d524f2ca60760ebcc30c1530477d78378b24d2253e2c6d2431324f48b" exitCode=0
Sep 05 00:04:16.314641 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:16.314477    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"
Sep 05 00:04:16.314641 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:16.314525    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt" event={"ID":"d8bdac91-7d4e-48a9-a384-1eaf25625fd3","Type":"ContainerDied","Data":"7c81445d524f2ca60760ebcc30c1530477d78378b24d2253e2c6d2431324f48b"}
Sep 05 00:04:16.314641 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:16.314573    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt" event={"ID":"d8bdac91-7d4e-48a9-a384-1eaf25625fd3","Type":"ContainerDied","Data":"1abf3a7efbd430429753c05983cf40014af19df5ac0fbc67117995f90e401b7a"}
Sep 05 00:04:16.314641 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:16.314595    2557 scope.go:117] "RemoveContainer" containerID="7c81445d524f2ca60760ebcc30c1530477d78378b24d2253e2c6d2431324f48b"
Sep 05 00:04:16.323341 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:16.323326    2557 scope.go:117] "RemoveContainer" containerID="7c81445d524f2ca60760ebcc30c1530477d78378b24d2253e2c6d2431324f48b"
Sep 05 00:04:16.323601 ip-10-0-142-236 kubenswrapper[2557]: E0905 00:04:16.323585    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c81445d524f2ca60760ebcc30c1530477d78378b24d2253e2c6d2431324f48b\": container with ID starting with 7c81445d524f2ca60760ebcc30c1530477d78378b24d2253e2c6d2431324f48b not found: ID does not exist" containerID="7c81445d524f2ca60760ebcc30c1530477d78378b24d2253e2c6d2431324f48b"
Sep 05 00:04:16.323638 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:16.323609    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c81445d524f2ca60760ebcc30c1530477d78378b24d2253e2c6d2431324f48b"} err="failed to get container status \"7c81445d524f2ca60760ebcc30c1530477d78378b24d2253e2c6d2431324f48b\": rpc error: code = NotFound desc = could not find container \"7c81445d524f2ca60760ebcc30c1530477d78378b24d2253e2c6d2431324f48b\": container with ID starting with 7c81445d524f2ca60760ebcc30c1530477d78378b24d2253e2c6d2431324f48b not found: ID does not exist"
Sep 05 00:04:16.330525 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:16.330472    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"]
Sep 05 00:04:16.334544 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:16.334524    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-88828d-98786f4c4-zlpkt"]
Sep 05 00:04:18.078762 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:18.078731    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8bdac91-7d4e-48a9-a384-1eaf25625fd3" path="/var/lib/kubelet/pods/d8bdac91-7d4e-48a9-a384-1eaf25625fd3/volumes"
Sep 05 00:04:18.322437 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:18.322406    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:04:22.122661 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:22.122630    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"]
Sep 05 00:04:22.123114 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:22.122827    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm" podUID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerName="istio-proxy" containerID="cri-o://b000c5065bb98a9cce59e7b175e5b9cf55d1f0bc0164a34d8eff82f6f172764e" gracePeriod=30
Sep 05 00:04:24.596816 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:24.596786    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 05 00:04:24.597208 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:24.596837    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm" podUID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 00:04:27.374132 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.374107    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 05 00:04:27.375989 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.375960    2557 generic.go:356] "Generic (PLEG): container finished" podID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerID="b000c5065bb98a9cce59e7b175e5b9cf55d1f0bc0164a34d8eff82f6f172764e" exitCode=0
Sep 05 00:04:27.376104 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.376004    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"
Sep 05 00:04:27.376104 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.376033    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm" event={"ID":"11d6c9bd-fae5-412a-b7fc-c3319e634b29","Type":"ContainerDied","Data":"b000c5065bb98a9cce59e7b175e5b9cf55d1f0bc0164a34d8eff82f6f172764e"}
Sep 05 00:04:27.376176 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.376108    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm" event={"ID":"11d6c9bd-fae5-412a-b7fc-c3319e634b29","Type":"ContainerDied","Data":"3fb91b7841fe86446fd9930a84d019fc008e7e92c2a2492dc543adfba648a050"}
Sep 05 00:04:27.376176 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.376132    2557 scope.go:117] "RemoveContainer" containerID="b000c5065bb98a9cce59e7b175e5b9cf55d1f0bc0164a34d8eff82f6f172764e"
Sep 05 00:04:27.384294 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.384279    2557 scope.go:117] "RemoveContainer" containerID="8017483328ccd8cd29163954e4174dc0bf12d46629aeda6873bc734be0b4ae44"
Sep 05 00:04:27.391784 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.391769    2557 scope.go:117] "RemoveContainer" containerID="b000c5065bb98a9cce59e7b175e5b9cf55d1f0bc0164a34d8eff82f6f172764e"
Sep 05 00:04:27.392079 ip-10-0-142-236 kubenswrapper[2557]: E0905 00:04:27.392059    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b000c5065bb98a9cce59e7b175e5b9cf55d1f0bc0164a34d8eff82f6f172764e\": container with ID starting with b000c5065bb98a9cce59e7b175e5b9cf55d1f0bc0164a34d8eff82f6f172764e not found: ID does not exist" containerID="b000c5065bb98a9cce59e7b175e5b9cf55d1f0bc0164a34d8eff82f6f172764e"
Sep 05 00:04:27.392156 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.392096    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b000c5065bb98a9cce59e7b175e5b9cf55d1f0bc0164a34d8eff82f6f172764e"} err="failed to get container status \"b000c5065bb98a9cce59e7b175e5b9cf55d1f0bc0164a34d8eff82f6f172764e\": rpc error: code = NotFound desc = could not find container \"b000c5065bb98a9cce59e7b175e5b9cf55d1f0bc0164a34d8eff82f6f172764e\": container with ID starting with b000c5065bb98a9cce59e7b175e5b9cf55d1f0bc0164a34d8eff82f6f172764e not found: ID does not exist"
Sep 05 00:04:27.392156 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.392120    2557 scope.go:117] "RemoveContainer" containerID="8017483328ccd8cd29163954e4174dc0bf12d46629aeda6873bc734be0b4ae44"
Sep 05 00:04:27.392405 ip-10-0-142-236 kubenswrapper[2557]: E0905 00:04:27.392379    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8017483328ccd8cd29163954e4174dc0bf12d46629aeda6873bc734be0b4ae44\": container with ID starting with 8017483328ccd8cd29163954e4174dc0bf12d46629aeda6873bc734be0b4ae44 not found: ID does not exist" containerID="8017483328ccd8cd29163954e4174dc0bf12d46629aeda6873bc734be0b4ae44"
Sep 05 00:04:27.392508 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.392419    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8017483328ccd8cd29163954e4174dc0bf12d46629aeda6873bc734be0b4ae44"} err="failed to get container status \"8017483328ccd8cd29163954e4174dc0bf12d46629aeda6873bc734be0b4ae44\": rpc error: code = NotFound desc = could not find container \"8017483328ccd8cd29163954e4174dc0bf12d46629aeda6873bc734be0b4ae44\": container with ID starting with 8017483328ccd8cd29163954e4174dc0bf12d46629aeda6873bc734be0b4ae44 not found: ID does not exist"
Sep 05 00:04:27.516581 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.516546    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istiod-ca-cert\") pod \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") "
Sep 05 00:04:27.516762 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.516597    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-workload-certs\") pod \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") "
Sep 05 00:04:27.516762 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.516648    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-podinfo\") pod \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") "
Sep 05 00:04:27.516762 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.516673    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-envoy\") pod \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") "
Sep 05 00:04:27.516762 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.516701    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-token\") pod \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") "
Sep 05 00:04:27.516762 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.516727    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-credential-socket\") pod \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") "
Sep 05 00:04:27.516762 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.516759    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-workload-socket\") pod \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") "
Sep 05 00:04:27.517112 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.516783    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-data\") pod \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") "
Sep 05 00:04:27.517112 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.516844    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-277fz\" (UniqueName: \"kubernetes.io/projected/11d6c9bd-fae5-412a-b7fc-c3319e634b29-kube-api-access-277fz\") pod \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\" (UID: \"11d6c9bd-fae5-412a-b7fc-c3319e634b29\") "
Sep 05 00:04:27.517112 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.517021    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11d6c9bd-fae5-412a-b7fc-c3319e634b29" (UID: "11d6c9bd-fae5-412a-b7fc-c3319e634b29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:04:27.517112 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.517065    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11d6c9bd-fae5-412a-b7fc-c3319e634b29" (UID: "11d6c9bd-fae5-412a-b7fc-c3319e634b29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 00:04:27.517112 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.517070    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11d6c9bd-fae5-412a-b7fc-c3319e634b29" (UID: "11d6c9bd-fae5-412a-b7fc-c3319e634b29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:04:27.517388 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.517243    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-credential-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:04:27.517388 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.517263    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istiod-ca-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:04:27.517388 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.517276    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-workload-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:04:27.517388 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.517247    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11d6c9bd-fae5-412a-b7fc-c3319e634b29" (UID: "11d6c9bd-fae5-412a-b7fc-c3319e634b29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:04:27.517388 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.517260    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-data" (OuterVolumeSpecName: "istio-data") pod "11d6c9bd-fae5-412a-b7fc-c3319e634b29" (UID: "11d6c9bd-fae5-412a-b7fc-c3319e634b29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:04:27.519012 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.518989    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d6c9bd-fae5-412a-b7fc-c3319e634b29-kube-api-access-277fz" (OuterVolumeSpecName: "kube-api-access-277fz") pod "11d6c9bd-fae5-412a-b7fc-c3319e634b29" (UID: "11d6c9bd-fae5-412a-b7fc-c3319e634b29"). InnerVolumeSpecName "kube-api-access-277fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:04:27.519127 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.519030    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11d6c9bd-fae5-412a-b7fc-c3319e634b29" (UID: "11d6c9bd-fae5-412a-b7fc-c3319e634b29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 00:04:27.519289 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.519258    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-token" (OuterVolumeSpecName: "istio-token") pod "11d6c9bd-fae5-412a-b7fc-c3319e634b29" (UID: "11d6c9bd-fae5-412a-b7fc-c3319e634b29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:04:27.519289 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.519279    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11d6c9bd-fae5-412a-b7fc-c3319e634b29" (UID: "11d6c9bd-fae5-412a-b7fc-c3319e634b29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:04:27.618339 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.618312    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-277fz\" (UniqueName: \"kubernetes.io/projected/11d6c9bd-fae5-412a-b7fc-c3319e634b29-kube-api-access-277fz\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:04:27.618339 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.618336    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-podinfo\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:04:27.618481 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.618345    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-envoy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:04:27.618481 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.618353    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:04:27.618481 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.618361    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-workload-socket\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:04:27.618481 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.618369    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d6c9bd-fae5-412a-b7fc-c3319e634b29-istio-data\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:04:27.698833 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.698809    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"]
Sep 05 00:04:27.702875 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:27.702850    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-88828d-openshift-default-76f9bf6678-cmvmm"]
Sep 05 00:04:28.079065 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:28.079032    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" path="/var/lib/kubelet/pods/11d6c9bd-fae5-412a-b7fc-c3319e634b29/volumes"
Sep 05 00:04:28.117375 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:04:28.117349    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:05:46.120259 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:05:46.120221    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:05:58.469740 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:05:58.469708    2557 scope.go:117] "RemoveContainer" containerID="e991389f7e195919ea1a27aac21adc7b9cd2a0ae74320472da341efa456e208d"
Sep 05 00:06:13.714524 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:06:13.714483    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:06:20.511149 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:06:20.511112    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:06:27.509889 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:06:27.509853    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:06:36.119527 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:06:36.119492    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:06:53.519199 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:06:53.519167    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:06:58.183527 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:06:58.183495    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 05 00:06:58.188975 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:06:58.188930    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w6m5m_5105066c-f4b4-450a-944c-e7dcd6964851/ovn-acl-logging/0.log"
Sep 05 00:06:58.489625 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:06:58.489596    2557 scope.go:117] "RemoveContainer" containerID="75164d36fe1a0b830cc817ead9d3c152406afb0481e506253f3058c09a4c3207"
Sep 05 00:06:58.500544 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:06:58.500521    2557 scope.go:117] "RemoveContainer" containerID="c46ede07f728ff2ed0ccbe2fe4685b7f03b201e108c10933744c5fecf1c0b727"
Sep 05 00:08:21.114138 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:21.114048    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:08:29.214249 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:29.214211    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:08:34.919345 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:34.919305    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:08:51.921287 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:51.921252    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:08:55.216091 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.216058    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"]
Sep 05 00:08:55.216560 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.216443    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8bdac91-7d4e-48a9-a384-1eaf25625fd3" containerName="payload-pre-processing"
Sep 05 00:08:55.216560 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.216456    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8bdac91-7d4e-48a9-a384-1eaf25625fd3" containerName="payload-pre-processing"
Sep 05 00:08:55.216560 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.216467    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cb172db-b007-4250-a82f-6a2603d70d68" containerName="revoke-keys"
Sep 05 00:08:55.216560 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.216473    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cb172db-b007-4250-a82f-6a2603d70d68" containerName="revoke-keys"
Sep 05 00:08:55.216560 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.216484    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerName="istio-proxy"
Sep 05 00:08:55.216560 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.216490    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerName="istio-proxy"
Sep 05 00:08:55.216560 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.216502    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerName="istio-proxy"
Sep 05 00:08:55.216560 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.216511    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerName="istio-proxy"
Sep 05 00:08:55.216560 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.216562    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerName="istio-proxy"
Sep 05 00:08:55.217069 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.216570    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8bdac91-7d4e-48a9-a384-1eaf25625fd3" containerName="payload-pre-processing"
Sep 05 00:08:55.217069 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.216585    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cb172db-b007-4250-a82f-6a2603d70d68" containerName="revoke-keys"
Sep 05 00:08:55.219520 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.219500    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.221848 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.221827    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-4ef033b4-kserve-self-signed-certs\""
Sep 05 00:08:55.229378 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.229358    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"]
Sep 05 00:08:55.414746 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.414712    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-home\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.414746 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.414748    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-tmp-dir\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.414955 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.414840    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-dshm\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.414955 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.414859    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lhhh\" (UniqueName: \"kubernetes.io/projected/8e3e1431-2a5b-46d4-94fb-74358f9338ac-kube-api-access-7lhhh\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.414955 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.414877    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e3e1431-2a5b-46d4-94fb-74358f9338ac-tls-certs\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.415071 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.414959    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-model-cache\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.516380 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.516306    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-dshm\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.516380 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.516341    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lhhh\" (UniqueName: \"kubernetes.io/projected/8e3e1431-2a5b-46d4-94fb-74358f9338ac-kube-api-access-7lhhh\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.516380 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.516362    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e3e1431-2a5b-46d4-94fb-74358f9338ac-tls-certs\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.516644 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.516393    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-model-cache\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.516644 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.516460    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-home\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.516644 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.516486    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-tmp-dir\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.516853 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.516832    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-model-cache\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.516934 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.516890    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-tmp-dir\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.517013 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.516923    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-home\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.518599 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.518573    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-dshm\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.518860 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.518843    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e3e1431-2a5b-46d4-94fb-74358f9338ac-tls-certs\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.523459 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.523438    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lhhh\" (UniqueName: \"kubernetes.io/projected/8e3e1431-2a5b-46d4-94fb-74358f9338ac-kube-api-access-7lhhh\") pod \"e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") " pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.530310 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.530289    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:08:55.800990 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:55.800901    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"]
Sep 05 00:08:55.803862 ip-10-0-142-236 kubenswrapper[2557]: W0905 00:08:55.803830    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e3e1431_2a5b_46d4_94fb_74358f9338ac.slice/crio-21770f532b33eda247c92a6252f24294ff0bbfaa5ae03d009a2cec2f323b7a73 WatchSource:0}: Error finding container 21770f532b33eda247c92a6252f24294ff0bbfaa5ae03d009a2cec2f323b7a73: Status 404 returned error can't find the container with id 21770f532b33eda247c92a6252f24294ff0bbfaa5ae03d009a2cec2f323b7a73
Sep 05 00:08:56.759356 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:56.759316    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t" event={"ID":"8e3e1431-2a5b-46d4-94fb-74358f9338ac","Type":"ContainerStarted","Data":"c410ad0febc1b1402028eb854abd855eae7e58e0f4a85a8006e80f8b5173f72f"}
Sep 05 00:08:56.759722 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:56.759367    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t" event={"ID":"8e3e1431-2a5b-46d4-94fb-74358f9338ac","Type":"ContainerStarted","Data":"21770f532b33eda247c92a6252f24294ff0bbfaa5ae03d009a2cec2f323b7a73"}
Sep 05 00:08:56.775974 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:56.775910    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t" podStartSLOduration=1.775894702 podStartE2EDuration="1.775894702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 00:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 00:08:56.774360923 +0000 UTC m=+1919.192179242" watchObservedRunningTime="2026-09-05 00:08:56.775894702 +0000 UTC m=+1919.193713020"
Sep 05 00:08:58.532785 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:58.532752    2557 scope.go:117] "RemoveContainer" containerID="0d966e1cf7e65ddc7e06af7e40431077733e69d80cd86d97fbbde3bf730c9d08"
Sep 05 00:08:58.541217 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:58.541195    2557 scope.go:117] "RemoveContainer" containerID="5f1f98f7522c31908fa9a822fbabbba1d3dd9f1d89fbabb06e256a4d89e1c0ff"
Sep 05 00:08:59.814120 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:08:59.814087    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:09:05.531085 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:05.531046    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:09:05.531510 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:05.531099    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:09:05.543238 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:05.543218    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:09:05.814702 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:05.814631    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:09:26.897606 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:26.897570    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"]
Sep 05 00:09:29.118316 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:29.118278    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:09:36.322080 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:36.322050    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:09:41.919514 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:41.919457    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t" podUID="8e3e1431-2a5b-46d4-94fb-74358f9338ac" containerName="main" containerID="cri-o://c410ad0febc1b1402028eb854abd855eae7e58e0f4a85a8006e80f8b5173f72f" gracePeriod=45
Sep 05 00:09:42.157183 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.157161    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:09:42.267204 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.267169    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-dshm\") pod \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") "
Sep 05 00:09:42.267204 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.267213    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-model-cache\") pod \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") "
Sep 05 00:09:42.267402 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.267258    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-home\") pod \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") "
Sep 05 00:09:42.267402 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.267306    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-tmp-dir\") pod \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") "
Sep 05 00:09:42.267402 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.267327    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e3e1431-2a5b-46d4-94fb-74358f9338ac-tls-certs\") pod \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") "
Sep 05 00:09:42.267402 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.267350    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lhhh\" (UniqueName: \"kubernetes.io/projected/8e3e1431-2a5b-46d4-94fb-74358f9338ac-kube-api-access-7lhhh\") pod \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\" (UID: \"8e3e1431-2a5b-46d4-94fb-74358f9338ac\") "
Sep 05 00:09:42.267614 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.267510    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e3e1431-2a5b-46d4-94fb-74358f9338ac" (UID: "8e3e1431-2a5b-46d4-94fb-74358f9338ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:09:42.267614 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.267538    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e3e1431-2a5b-46d4-94fb-74358f9338ac" (UID: "8e3e1431-2a5b-46d4-94fb-74358f9338ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:09:42.267614 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.267558    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-home" (OuterVolumeSpecName: "home") pod "8e3e1431-2a5b-46d4-94fb-74358f9338ac" (UID: "8e3e1431-2a5b-46d4-94fb-74358f9338ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:09:42.269274 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.269248    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-dshm" (OuterVolumeSpecName: "dshm") pod "8e3e1431-2a5b-46d4-94fb-74358f9338ac" (UID: "8e3e1431-2a5b-46d4-94fb-74358f9338ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 00:09:42.269370 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.269349    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3e1431-2a5b-46d4-94fb-74358f9338ac-kube-api-access-7lhhh" (OuterVolumeSpecName: "kube-api-access-7lhhh") pod "8e3e1431-2a5b-46d4-94fb-74358f9338ac" (UID: "8e3e1431-2a5b-46d4-94fb-74358f9338ac"). InnerVolumeSpecName "kube-api-access-7lhhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 00:09:42.269370 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.269361    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e3e1431-2a5b-46d4-94fb-74358f9338ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e3e1431-2a5b-46d4-94fb-74358f9338ac" (UID: "8e3e1431-2a5b-46d4-94fb-74358f9338ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 00:09:42.367861 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.367836    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:09:42.367861 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.367858    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e3e1431-2a5b-46d4-94fb-74358f9338ac-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:09:42.368020 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.367868    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lhhh\" (UniqueName: \"kubernetes.io/projected/8e3e1431-2a5b-46d4-94fb-74358f9338ac-kube-api-access-7lhhh\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:09:42.368020 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.367879    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:09:42.368020 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.367887    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:09:42.368020 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:42.367894    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e3e1431-2a5b-46d4-94fb-74358f9338ac-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\""
Sep 05 00:09:43.000917 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:43.000874    2557 generic.go:356] "Generic (PLEG): container finished" podID="8e3e1431-2a5b-46d4-94fb-74358f9338ac" containerID="c410ad0febc1b1402028eb854abd855eae7e58e0f4a85a8006e80f8b5173f72f" exitCode=0
Sep 05 00:09:43.001361 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:43.000931    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"
Sep 05 00:09:43.001361 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:43.000975    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t" event={"ID":"8e3e1431-2a5b-46d4-94fb-74358f9338ac","Type":"ContainerDied","Data":"c410ad0febc1b1402028eb854abd855eae7e58e0f4a85a8006e80f8b5173f72f"}
Sep 05 00:09:43.001361 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:43.001021    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t" event={"ID":"8e3e1431-2a5b-46d4-94fb-74358f9338ac","Type":"ContainerDied","Data":"21770f532b33eda247c92a6252f24294ff0bbfaa5ae03d009a2cec2f323b7a73"}
Sep 05 00:09:43.001361 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:43.001038    2557 scope.go:117] "RemoveContainer" containerID="c410ad0febc1b1402028eb854abd855eae7e58e0f4a85a8006e80f8b5173f72f"
Sep 05 00:09:43.010029 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:43.010011    2557 scope.go:117] "RemoveContainer" containerID="c410ad0febc1b1402028eb854abd855eae7e58e0f4a85a8006e80f8b5173f72f"
Sep 05 00:09:43.010277 ip-10-0-142-236 kubenswrapper[2557]: E0905 00:09:43.010258    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c410ad0febc1b1402028eb854abd855eae7e58e0f4a85a8006e80f8b5173f72f\": container with ID starting with c410ad0febc1b1402028eb854abd855eae7e58e0f4a85a8006e80f8b5173f72f not found: ID does not exist" containerID="c410ad0febc1b1402028eb854abd855eae7e58e0f4a85a8006e80f8b5173f72f"
Sep 05 00:09:43.010323 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:43.010285    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c410ad0febc1b1402028eb854abd855eae7e58e0f4a85a8006e80f8b5173f72f"} err="failed to get container status \"c410ad0febc1b1402028eb854abd855eae7e58e0f4a85a8006e80f8b5173f72f\": rpc error: code = NotFound desc = could not find container \"c410ad0febc1b1402028eb854abd855eae7e58e0f4a85a8006e80f8b5173f72f\": container with ID starting with c410ad0febc1b1402028eb854abd855eae7e58e0f4a85a8006e80f8b5173f72f not found: ID does not exist"
Sep 05 00:09:43.021027 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:43.021002    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"]
Sep 05 00:09:43.024709 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:43.024688    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-4ef033b4-kserve-78f5d9b74c-8f77t"]
Sep 05 00:09:44.078414 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:44.078380    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e3e1431-2a5b-46d4-94fb-74358f9338ac" path="/var/lib/kubelet/pods/8e3e1431-2a5b-46d4-94fb-74358f9338ac/volumes"
Sep 05 00:09:45.017369 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:45.017333    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:09:51.811377 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:51.811292    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:09:58.566859 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:09:58.566816    2557 scope.go:117] "RemoveContainer" containerID="d7ecd50512e453102d78663a7c1324b47ec9de0d1e4b2468e33d1c436c3c82c0"
Sep 05 00:10:02.215123 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:02.215092    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:10:08.216865 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:08.213688    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:10:17.218979 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:17.218921    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-jcwjt"]
Sep 05 00:10:24.477627 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:24.477596    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-w7662_98446e0d-67db-497e-83f2-ecbb86cb2f30/server/0.log"
Sep 05 00:10:24.597613 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:24.597581    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4svvv_38d54642-90cf-4657-8eda-2e493f2dd61c/manager/0.log"
Sep 05 00:10:24.957712 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:24.957687    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b9496bc86-w7fwm_3068ccbb-0fca-4923-b3cb-e5731bfe1dfa/manager/0.log"
Sep 05 00:10:24.962349 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:24.962329    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b9496bc86-w7fwm_3068ccbb-0fca-4923-b3cb-e5731bfe1dfa/copy-manifests/0.log"
Sep 05 00:10:26.923337 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:26.923304    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-jcwjt_fbc0ecf2-7717-4c7a-a285-20f4e70dce6f/limitador/0.log"
Sep 05 00:10:27.405715 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:27.405679    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z_9507b3c4-5133-4c67-ae0e-b6f7ce22d313/istio-proxy/0.log"
Sep 05 00:10:27.518632 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:27.518601    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j2lqk_f272cabc-d99d-4b08-beea-4aa119a495c5/discovery/0.log"
Sep 05 00:10:29.189210 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:29.189177    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-6tk2f_58da7198-ec0f-4a2f-a77e-ee68c4c34cd2/main/0.log"
Sep 05 00:10:34.520114 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.520078    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c2q7w/must-gather-ddplt"]
Sep 05 00:10:34.520702 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.520597    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e3e1431-2a5b-46d4-94fb-74358f9338ac" containerName="main"
Sep 05 00:10:34.520702 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.520618    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e3e1431-2a5b-46d4-94fb-74358f9338ac" containerName="main"
Sep 05 00:10:34.520702 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.520700    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e3e1431-2a5b-46d4-94fb-74358f9338ac" containerName="main"
Sep 05 00:10:34.520885 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.520713    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d6c9bd-fae5-412a-b7fc-c3319e634b29" containerName="istio-proxy"
Sep 05 00:10:34.523815 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.523793    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c2q7w/must-gather-ddplt"
Sep 05 00:10:34.526735 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.526713    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c2q7w\"/\"default-dockercfg-8v5v9\""
Sep 05 00:10:34.526893 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.526826    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c2q7w\"/\"openshift-service-ca.crt\""
Sep 05 00:10:34.527522 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.527500    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c2q7w\"/\"kube-root-ca.crt\""
Sep 05 00:10:34.532648 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.532627    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c2q7w/must-gather-ddplt"]
Sep 05 00:10:34.660236 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.660205    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/022b99ac-e23e-43ef-9dc3-8d14d0a90adc-must-gather-output\") pod \"must-gather-ddplt\" (UID: \"022b99ac-e23e-43ef-9dc3-8d14d0a90adc\") " pod="openshift-must-gather-c2q7w/must-gather-ddplt"
Sep 05 00:10:34.660342 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.660259    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncfz8\" (UniqueName: \"kubernetes.io/projected/022b99ac-e23e-43ef-9dc3-8d14d0a90adc-kube-api-access-ncfz8\") pod \"must-gather-ddplt\" (UID: \"022b99ac-e23e-43ef-9dc3-8d14d0a90adc\") " pod="openshift-must-gather-c2q7w/must-gather-ddplt"
Sep 05 00:10:34.761288 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.761252    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/022b99ac-e23e-43ef-9dc3-8d14d0a90adc-must-gather-output\") pod \"must-gather-ddplt\" (UID: \"022b99ac-e23e-43ef-9dc3-8d14d0a90adc\") " pod="openshift-must-gather-c2q7w/must-gather-ddplt"
Sep 05 00:10:34.761385 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.761310    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncfz8\" (UniqueName: \"kubernetes.io/projected/022b99ac-e23e-43ef-9dc3-8d14d0a90adc-kube-api-access-ncfz8\") pod \"must-gather-ddplt\" (UID: \"022b99ac-e23e-43ef-9dc3-8d14d0a90adc\") " pod="openshift-must-gather-c2q7w/must-gather-ddplt"
Sep 05 00:10:34.761634 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.761613    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/022b99ac-e23e-43ef-9dc3-8d14d0a90adc-must-gather-output\") pod \"must-gather-ddplt\" (UID: \"022b99ac-e23e-43ef-9dc3-8d14d0a90adc\") " pod="openshift-must-gather-c2q7w/must-gather-ddplt"
Sep 05 00:10:34.769293 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.769273    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncfz8\" (UniqueName: \"kubernetes.io/projected/022b99ac-e23e-43ef-9dc3-8d14d0a90adc-kube-api-access-ncfz8\") pod \"must-gather-ddplt\" (UID: \"022b99ac-e23e-43ef-9dc3-8d14d0a90adc\") " pod="openshift-must-gather-c2q7w/must-gather-ddplt"
Sep 05 00:10:34.834236 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:34.834187    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c2q7w/must-gather-ddplt"
Sep 05 00:10:35.110161 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:35.110129    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c2q7w/must-gather-ddplt"]
Sep 05 00:10:35.111785 ip-10-0-142-236 kubenswrapper[2557]: W0905 00:10:35.111755    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod022b99ac_e23e_43ef_9dc3_8d14d0a90adc.slice/crio-2cef1fc2f79d19c1488b5fa6b919b51adccdce2adc356c16ef000788e37ccf59 WatchSource:0}: Error finding container 2cef1fc2f79d19c1488b5fa6b919b51adccdce2adc356c16ef000788e37ccf59: Status 404 returned error can't find the container with id 2cef1fc2f79d19c1488b5fa6b919b51adccdce2adc356c16ef000788e37ccf59
Sep 05 00:10:35.113980 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:35.113966    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 00:10:35.262170 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:35.262142    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c2q7w/must-gather-ddplt" event={"ID":"022b99ac-e23e-43ef-9dc3-8d14d0a90adc","Type":"ContainerStarted","Data":"2cef1fc2f79d19c1488b5fa6b919b51adccdce2adc356c16ef000788e37ccf59"}
Sep 05 00:10:37.276531 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:37.276495    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c2q7w/must-gather-ddplt" event={"ID":"022b99ac-e23e-43ef-9dc3-8d14d0a90adc","Type":"ContainerStarted","Data":"5823de07094eb07f52d3318dfab7f63855b3ace95bf057e8546096f8a8987be3"}
Sep 05 00:10:37.276531 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:37.276538    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c2q7w/must-gather-ddplt" event={"ID":"022b99ac-e23e-43ef-9dc3-8d14d0a90adc","Type":"ContainerStarted","Data":"12e12464c61a2f01f6047078bf354b531d64d0e374984832e62e2b15f5785ff9"}
Sep 05 00:10:37.292295 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:37.292238    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c2q7w/must-gather-ddplt" podStartSLOduration=2.218640826 podStartE2EDuration="3.292218344s" totalImagesPullingTime="1.073577518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 00:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 00:10:37.290264218 +0000 UTC m=+2019.708082537" watchObservedRunningTime="2026-09-05 00:10:37.292218344 +0000 UTC m=+2019.710036665"
Sep 05 00:10:37.775587 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:37.775555    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sj4n7_9dc959dc-ab62-40f7-81ea-e62e4aafed21/global-pull-secret-syncer/0.log"
Sep 05 00:10:37.934661 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:37.934629    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rkpz9_e356b35c-5280-4e59-9cd2-303af05624a3/konnectivity-agent/0.log"
Sep 05 00:10:38.020643 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:38.020603    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-236.ec2.internal_3170cdb077432b791941250ec565815c/haproxy/0.log"
Sep 05 00:10:42.892584 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:42.892553    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-jcwjt_fbc0ecf2-7717-4c7a-a285-20f4e70dce6f/limitador/0.log"
Sep 05 00:10:44.542494 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:44.542458    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-hk5cq_fc699e03-b224-47ec-92dd-c1d4f41add41/kube-state-metrics/0.log"
Sep 05 00:10:44.559491 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:44.559387    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-hk5cq_fc699e03-b224-47ec-92dd-c1d4f41add41/kube-rbac-proxy-main/0.log"
Sep 05 00:10:44.578780 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:44.578745    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-hk5cq_fc699e03-b224-47ec-92dd-c1d4f41add41/kube-rbac-proxy-self/0.log"
Sep 05 00:10:44.612470 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:44.612437    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-68cb67bcf5-j584m_88735e65-4065-495a-971e-e617a52e52de/metrics-server/0.log"
Sep 05 00:10:44.637005 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:44.636971    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-8km96_25b90e0e-65d5-4510-a013-8c4795de5065/monitoring-plugin/0.log"
Sep 05 00:10:44.813860 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:44.813762    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zf9mx_9717ba4d-cf89-4e64-a7ce-4d119569e7f0/node-exporter/0.log"
Sep 05 00:10:44.832807 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:44.832777    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zf9mx_9717ba4d-cf89-4e64-a7ce-4d119569e7f0/kube-rbac-proxy/0.log"
Sep 05 00:10:44.867818 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:44.867778    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zf9mx_9717ba4d-cf89-4e64-a7ce-4d119569e7f0/init-textfile/0.log"
Sep 05 00:10:44.893510 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:44.893475    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-z49js_9aa8ad72-aca0-4abb-9e56-efc967a6e1da/kube-rbac-proxy-main/0.log"
Sep 05 00:10:44.915228 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:44.915197    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-z49js_9aa8ad72-aca0-4abb-9e56-efc967a6e1da/kube-rbac-proxy-self/0.log"
Sep 05 00:10:44.940594 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:44.940558    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-z49js_9aa8ad72-aca0-4abb-9e56-efc967a6e1da/openshift-state-metrics/0.log"
Sep 05 00:10:45.339515 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:45.339481    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fffc77b74-rjpc7_9bbca191-e2ec-4175-bc12-ebee023e8ba4/thanos-query/0.log"
Sep 05 00:10:45.359678 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:45.359644    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fffc77b74-rjpc7_9bbca191-e2ec-4175-bc12-ebee023e8ba4/kube-rbac-proxy-web/0.log"
Sep 05 00:10:45.379756 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:45.379727    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fffc77b74-rjpc7_9bbca191-e2ec-4175-bc12-ebee023e8ba4/kube-rbac-proxy/0.log"
Sep 05 00:10:45.400854 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:45.400752    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fffc77b74-rjpc7_9bbca191-e2ec-4175-bc12-ebee023e8ba4/prom-label-proxy/0.log"
Sep 05 00:10:45.422138 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:45.422109    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fffc77b74-rjpc7_9bbca191-e2ec-4175-bc12-ebee023e8ba4/kube-rbac-proxy-rules/0.log"
Sep 05 00:10:45.442653 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:45.442595    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fffc77b74-rjpc7_9bbca191-e2ec-4175-bc12-ebee023e8ba4/kube-rbac-proxy-metrics/0.log"
Sep 05 00:10:47.543049 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.543008    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"]
Sep 05 00:10:47.547510 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.547484    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.557206 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.556830    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"]
Sep 05 00:10:47.588716 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.588630    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fbaa30b-7f9c-4483-9a0a-fa689025f4fd-podres\") pod \"perf-node-gather-daemonset-pzdl8\" (UID: \"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.588716 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.588716    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fbaa30b-7f9c-4483-9a0a-fa689025f4fd-lib-modules\") pod \"perf-node-gather-daemonset-pzdl8\" (UID: \"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.588996 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.588815    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gbht\" (UniqueName: \"kubernetes.io/projected/4fbaa30b-7f9c-4483-9a0a-fa689025f4fd-kube-api-access-4gbht\") pod \"perf-node-gather-daemonset-pzdl8\" (UID: \"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.588996 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.588873    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fbaa30b-7f9c-4483-9a0a-fa689025f4fd-proc\") pod \"perf-node-gather-daemonset-pzdl8\" (UID: \"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.588996 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.588921    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fbaa30b-7f9c-4483-9a0a-fa689025f4fd-sys\") pod \"perf-node-gather-daemonset-pzdl8\" (UID: \"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.690129 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.690090    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fbaa30b-7f9c-4483-9a0a-fa689025f4fd-sys\") pod \"perf-node-gather-daemonset-pzdl8\" (UID: \"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.690320 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.690172    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fbaa30b-7f9c-4483-9a0a-fa689025f4fd-podres\") pod \"perf-node-gather-daemonset-pzdl8\" (UID: \"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.690320 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.690211    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fbaa30b-7f9c-4483-9a0a-fa689025f4fd-sys\") pod \"perf-node-gather-daemonset-pzdl8\" (UID: \"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.690320 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.690221    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fbaa30b-7f9c-4483-9a0a-fa689025f4fd-lib-modules\") pod \"perf-node-gather-daemonset-pzdl8\" (UID: \"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.690509 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.690323    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fbaa30b-7f9c-4483-9a0a-fa689025f4fd-podres\") pod \"perf-node-gather-daemonset-pzdl8\" (UID: \"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.690509 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.690343    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gbht\" (UniqueName: \"kubernetes.io/projected/4fbaa30b-7f9c-4483-9a0a-fa689025f4fd-kube-api-access-4gbht\") pod \"perf-node-gather-daemonset-pzdl8\" (UID: \"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.690509 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.690383    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fbaa30b-7f9c-4483-9a0a-fa689025f4fd-lib-modules\") pod \"perf-node-gather-daemonset-pzdl8\" (UID: \"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.690509 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.690390    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fbaa30b-7f9c-4483-9a0a-fa689025f4fd-proc\") pod \"perf-node-gather-daemonset-pzdl8\" (UID: \"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.690509 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.690442    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fbaa30b-7f9c-4483-9a0a-fa689025f4fd-proc\") pod \"perf-node-gather-daemonset-pzdl8\" (UID: \"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.699890 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.699839    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gbht\" (UniqueName: \"kubernetes.io/projected/4fbaa30b-7f9c-4483-9a0a-fa689025f4fd-kube-api-access-4gbht\") pod \"perf-node-gather-daemonset-pzdl8\" (UID: \"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd\") " pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:47.861415 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:47.861323    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:48.151366 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:48.151312    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"]
Sep 05 00:10:48.154252 ip-10-0-142-236 kubenswrapper[2557]: W0905 00:10:48.154215    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4fbaa30b_7f9c_4483_9a0a_fa689025f4fd.slice/crio-8b4a40f3050925f0f2777d49ee1c8735cf66356cd0709684734f0d25284463c8 WatchSource:0}: Error finding container 8b4a40f3050925f0f2777d49ee1c8735cf66356cd0709684734f0d25284463c8: Status 404 returned error can't find the container with id 8b4a40f3050925f0f2777d49ee1c8735cf66356cd0709684734f0d25284463c8
Sep 05 00:10:48.363949 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:48.363897    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8" event={"ID":"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd","Type":"ContainerStarted","Data":"520fe5e864da4090c47373b098a53aab4a0514749fcabc4af1120db20a2d27cd"}
Sep 05 00:10:48.364115 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:48.363977    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8" event={"ID":"4fbaa30b-7f9c-4483-9a0a-fa689025f4fd","Type":"ContainerStarted","Data":"8b4a40f3050925f0f2777d49ee1c8735cf66356cd0709684734f0d25284463c8"}
Sep 05 00:10:48.364805 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:48.364779    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:48.382877 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:48.382818    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8" podStartSLOduration=1.382797911 podStartE2EDuration="1.382797911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 00:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 00:10:48.378640015 +0000 UTC m=+2030.796458335" watchObservedRunningTime="2026-09-05 00:10:48.382797911 +0000 UTC m=+2030.800616230"
Sep 05 00:10:49.613475 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:49.613447    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-crz7n_d83926d9-914f-4bff-99ba-6ba74b32ada5/dns/0.log"
Sep 05 00:10:49.631865 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:49.631842    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-crz7n_d83926d9-914f-4bff-99ba-6ba74b32ada5/kube-rbac-proxy/0.log"
Sep 05 00:10:49.765032 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:49.764996    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-brgnz_f60b1cfd-03dc-43f3-9cce-a0bbffbb4226/dns-node-resolver/0.log"
Sep 05 00:10:50.288860 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:50.288828    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29809440-tfddf_40895fc7-36d0-424f-bcf9-0d1c5c5eb7ed/image-pruner/0.log"
Sep 05 00:10:50.331116 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:50.331085    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-766d6d64dc-7h6zs_69786b86-8ed4-44a2-805a-3021d4a5dc6f/registry/0.log"
Sep 05 00:10:50.397754 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:50.397727    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wkxtm_d6a5b60d-4a14-49cc-8be4-f86989c0bd88/node-ca/0.log"
Sep 05 00:10:51.273044 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:51.273010    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z_9507b3c4-5133-4c67-ae0e-b6f7ce22d313/istio-proxy/0.log"
Sep 05 00:10:51.852024 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:51.851985    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j2lqk_f272cabc-d99d-4b08-beea-4aa119a495c5/discovery/0.log"
Sep 05 00:10:52.886225 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:52.886177    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kh7ws_9f4ed71e-4b2e-4672-b7b8-832b3f4993de/serve-healthcheck-canary/0.log"
Sep 05 00:10:53.446995 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:53.446964    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8th6_514da723-8019-4825-a454-a1d56c30be6f/kube-rbac-proxy/0.log"
Sep 05 00:10:53.466542 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:53.466513    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8th6_514da723-8019-4825-a454-a1d56c30be6f/exporter/0.log"
Sep 05 00:10:53.486624 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:53.486593    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8th6_514da723-8019-4825-a454-a1d56c30be6f/extractor/0.log"
Sep 05 00:10:55.386918 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:55.386887    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c2q7w/perf-node-gather-daemonset-pzdl8"
Sep 05 00:10:55.819961 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:55.819906    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-w7662_98446e0d-67db-497e-83f2-ecbb86cb2f30/server/0.log"
Sep 05 00:10:55.952285 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:55.952248    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4svvv_38d54642-90cf-4657-8eda-2e493f2dd61c/manager/0.log"
Sep 05 00:10:56.191961 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:56.191566    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b9496bc86-w7fwm_3068ccbb-0fca-4923-b3cb-e5731bfe1dfa/manager/0.log"
Sep 05 00:10:56.207723 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:56.207695    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b9496bc86-w7fwm_3068ccbb-0fca-4923-b3cb-e5731bfe1dfa/copy-manifests/0.log"
Sep 05 00:10:57.635461 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:57.635432    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58b866555c-m5t4p_662aa4c8-4bd8-4817-82c9-fcc85b0d9833/manager/0.log"
Sep 05 00:10:57.662642 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:57.662602    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-bxs66_1fc2d476-f685-46ba-9dc0-feb88ddea0e9/openshift-lws-operator/0.log"
Sep 05 00:10:58.227592 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:58.227552    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z_9507b3c4-5133-4c67-ae0e-b6f7ce22d313/istio-proxy/0.log"
Sep 05 00:10:58.598102 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:58.596148    2557 scope.go:117] "RemoveContainer" containerID="5eabde5b40638a52519aef1d8029535bf981eb8c6f91e1398c2fffab0521b91b"
Sep 05 00:10:58.862913 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:10:58.862829    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j2lqk_f272cabc-d99d-4b08-beea-4aa119a495c5/discovery/0.log"
Sep 05 00:11:00.433849 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:11:00.433820    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cf6x8z_9507b3c4-5133-4c67-ae0e-b6f7ce22d313/istio-proxy/0.log"
Sep 05 00:11:00.988849 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:11:00.988772    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j2lqk_f272cabc-d99d-4b08-beea-4aa119a495c5/discovery/0.log"
Sep 05 00:11:04.412841 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:11:04.412816    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wg92b_740e83e1-71de-4cca-ba79-614a1f20be97/kube-multus-additional-cni-plugins/0.log"
Sep 05 00:11:04.434906 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:11:04.434882    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wg92b_740e83e1-71de-4cca-ba79-614a1f20be97/egress-router-binary-copy/0.log"
Sep 05 00:11:04.454832 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:11:04.454809    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wg92b_740e83e1-71de-4cca-ba79-614a1f20be97/cni-plugins/0.log"
Sep 05 00:11:04.478855 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:11:04.478828    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wg92b_740e83e1-71de-4cca-ba79-614a1f20be97/bond-cni-plugin/0.log"
Sep 05 00:11:04.498307 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:11:04.498287    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wg92b_740e83e1-71de-4cca-ba79-614a1f20be97/routeoverride-cni/0.log"
Sep 05 00:11:04.517274 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:11:04.517252    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wg92b_740e83e1-71de-4cca-ba79-614a1f20be97/whereabouts-cni-bincopy/0.log"
Sep 05 00:11:04.536480 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:11:04.536460    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wg92b_740e83e1-71de-4cca-ba79-614a1f20be97/whereabouts-cni/0.log"
Sep 05 00:11:04.640546 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:11:04.640450    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jjnzw_cfd119fb-c2f0-458e-819e-f2de497ef5dd/kube-multus/0.log"
Sep 05 00:11:04.799987 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:11:04.799956    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rd7qq_ca3463da-4922-4b81-a71e-8769f4a90494/network-metrics-daemon/0.log"
Sep 05 00:11:04.815859 ip-10-0-142-236 kubenswrapper[2557]: I0905 00:11:04.815824    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rd7qq_ca3463da-4922-4b81-a71e-8769f4a90494/kube-rbac-proxy/0.log"