Sep 09 15:49:32.835867 ip-10-0-133-93 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 15:49:32.835882 ip-10-0-133-93 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 15:49:32.835892 ip-10-0-133-93 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 15:49:32.836102 ip-10-0-133-93 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 15:49:42.852211 ip-10-0-133-93 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 15:49:42.852229 ip-10-0-133-93 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7e0d492fb8624b59906b37ac37840624 --
Sep 09 15:51:33.045873 ip-10-0-133-93 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 15:51:33.500437 ip-10-0-133-93 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 15:51:33.500437 ip-10-0-133-93 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 15:51:33.500437 ip-10-0-133-93 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 15:51:33.500437 ip-10-0-133-93 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 15:51:33.500437 ip-10-0-133-93 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 15:51:33.503302 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.503202    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 15:51:33.508267 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508245    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:51:33.508267 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508262    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:51:33.508267 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508265    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:51:33.508267 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508271    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508275    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508280    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508284    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508286    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508290    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508292    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508295    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508301    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508304    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508308    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508311    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508315    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508319    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508322    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508324    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508328    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508330    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508333    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:51:33.508435 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508336    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508340    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508343    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508345    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508348    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508351    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508353    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508356    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508358    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508361    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508370    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508372    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508375    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508377    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508380    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508384    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508396    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508399    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508404    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508407    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:51:33.508912 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508410    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508414    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508417    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508419    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508422    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508425    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508427    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508430    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508433    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508435    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508438    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508441    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508444    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508446    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508449    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508452    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508455    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508457    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508460    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508462    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:51:33.509401 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508465    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508467    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508470    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508472    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508476    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508479    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508483    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508486    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508488    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508492    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508495    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508498    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508501    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508504    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508506    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508509    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508511    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508514    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508516    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:51:33.509906 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508519    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508521    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508524    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508527    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508529    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508532    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508535    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508537    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508540    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508542    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508545    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508548    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508550    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508553    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508556    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508558    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508561    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.508563    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510401    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510409    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:51:33.510492 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510413    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510416    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510419    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510422    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510424    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510427    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510430    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510433    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510436    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510438    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510441    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510443    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510446    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510448    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510452    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510455    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510458    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510461    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510463    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510466    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:51:33.511001 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510469    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510471    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510474    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510477    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510480    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510482    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510486    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510490    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510493    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510495    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510498    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510501    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510504    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510506    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510509    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510512    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510514    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510517    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510522    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:51:33.511521 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510525    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510530    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510533    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510536    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510539    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510542    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510545    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510548    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510551    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510554    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510557    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510560    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510563    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510565    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510568    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510571    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510573    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510576    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510579    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:51:33.512011 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510582    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510584    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510601    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510604    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510607    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510610    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510612    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510615    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510617    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510620    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510622    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510625    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510628    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510631    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510633    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510636    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510639    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510641    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510644    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510646    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:51:33.512486 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510649    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510652    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510657    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510661    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510664    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510667    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510670    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510672    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510675    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510678    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510680    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510683    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510686    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510688    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510691    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510693    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510696    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510698    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.510701    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511079    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:51:33.513004 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511084    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511086    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511089    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511092    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511094    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511098    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511101    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511103    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511106    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511109    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511112    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511114    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511119    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511122    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511125    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511128    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511131    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511134    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511137    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:51:33.513488 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511140    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511143    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511145    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511148    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511151    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511153    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511156    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511159    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511162    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511165    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511167    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511170    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511172    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511175    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511177    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511180    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511183    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511186    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511189    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:51:33.513965 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511192    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511195    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511197    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511200    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511203    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511205    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511208    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511212    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511215    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511218    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511221    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511223    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511226    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511229    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511231    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511234    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511237    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511240    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511243    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511246    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:51:33.514426 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511249    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511252    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511255    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511258    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511260    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511263    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511265    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511268    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511271    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511274    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511277    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511281    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511284    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511287    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511290    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511292    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511294    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511297    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511299    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511302    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:51:33.514926 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511304    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511306    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511309    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511311    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511314    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511316    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511318    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511321    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511324    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511329    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511332    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511334    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511337    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511340    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511343    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511345    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511348    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511350    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511353    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.511355    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:51:33.515447 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511429    2556 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511490    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511498    2556 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511503    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511507    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511510    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511516    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511520    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511524    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511527    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511531    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511534    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511537    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511540    2556 flags.go:64] FLAG: --cgroup-root=""
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511543    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511546    2556 flags.go:64] FLAG: --client-ca-file=""
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511549    2556 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511552    2556 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511556    2556 flags.go:64] FLAG: --cluster-domain=""
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511559    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511562    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511567    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511570    2556 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511574    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 15:51:33.515937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511577    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511581    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511584    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511600    2556 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511604    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511607    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511610    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511613    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511617    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511620    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511623    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511626    2556 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511629    2556 flags.go:64] FLAG: --enable-server="true"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511632    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511636    2556 flags.go:64] FLAG: --event-burst="100"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511640    2556 flags.go:64] FLAG: --event-qps="50"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511643    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511647    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511649    2556 flags.go:64] FLAG: --eviction-hard=""
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511653    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511656    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511659    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511662    2556 flags.go:64] FLAG: --eviction-soft=""
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511665    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511667    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 15:51:33.516507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511670    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511673    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511676    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511679    2556 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511683    2556 flags.go:64] FLAG: --feature-gates=""
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511687    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511690    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511693    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511696    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511699    2556 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511702    2556 flags.go:64] FLAG: --help="false"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511705    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-133-93.ec2.internal"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511709    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511712    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511714    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511718    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511721    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511724    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511727    2556 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511730    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511733    2556 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511735    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511738    2556 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511742    2556 flags.go:64] FLAG: --kube-reserved=""
Sep 09 15:51:33.517139 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511745    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511748    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511751    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511754    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511756    2556 flags.go:64] FLAG: --lock-file=""
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511759    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511762    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511765    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511770    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511773    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511776    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511778    2556 flags.go:64] FLAG: --logging-format="text"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511782    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511786    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511789    2556 flags.go:64] FLAG: --manifest-url=""
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511791    2556 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511796    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511802    2556 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511806    2556 flags.go:64] FLAG: --max-pods="110"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511809    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511812    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511816    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511819    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511822    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511825    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 15:51:33.517781 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511828    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511835    2556 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511838    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511841    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511844    2556 flags.go:64] FLAG: --pod-cidr=""
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511848    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511853    2556 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511857    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511860    2556 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511863    2556 flags.go:64] FLAG: --port="10250"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511867    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511870    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eb1913329a578c82"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511873    2556 flags.go:64] FLAG: --qos-reserved=""
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511876    2556 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511878    2556 flags.go:64] FLAG: --register-node="true"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511881    2556 flags.go:64] FLAG: --register-with-taints=""
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511885    2556 flags.go:64] FLAG: --registry-burst="10"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511888    2556 flags.go:64] FLAG: --registry-qps="5"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511891    2556 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511896    2556 flags.go:64] FLAG: --reserved-memory=""
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511900    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511903    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511905    2556 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511908    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511911    2556 flags.go:64] FLAG: --runonce="false"
Sep 09 15:51:33.518404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511916    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511920    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511923    2556 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511926    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511928    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511932    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511935    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511938    2556 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511941    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511944    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511946    2556 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511949    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511952    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511955    2556 flags.go:64] FLAG: --system-cgroups=""
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511958    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511964    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511966    2556 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511969    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511973    2556 flags.go:64] FLAG: --tls-min-version=""
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511976    2556 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511979    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511981    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511984    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511987    2556 flags.go:64] FLAG: --v="2"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511991    2556 flags.go:64] FLAG: --version="false"
Sep 09 15:51:33.519080 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.511995    2556 flags.go:64] FLAG: --vmodule=""
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.512000    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.512003    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512121    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512126    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512129    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512133    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512137    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512140    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512143    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512146    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512149    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512151    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512154    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512156    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512159    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512161    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512164    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512167    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512169    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:51:33.519709 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512172    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512175    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512177    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512181    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512183    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512186    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512189    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512192    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512194    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512196    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512199    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512203    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512206    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512210    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512214    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512216    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512219    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512222    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512225    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:51:33.520199 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512230    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512232    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512235    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512237    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512240    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512243    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512245    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512247    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512250    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512253    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512255    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512258    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512260    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512263    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512265    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512268    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512271    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512274    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512277    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512280    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:51:33.520785 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512282    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512285    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512287    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512290    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512292    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512295    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512298    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512300    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512303    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512305    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512308    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512310    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512314    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512316    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512319    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512323    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512326    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512329    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512332    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512334    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:51:33.521550 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512337    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512339    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512342    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512344    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512347    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512349    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512352    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512354    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512357    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512359    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512362    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512365    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512367    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512370    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512372    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512375    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512377    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512380    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512383    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512392    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:51:33.522275 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512396    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:51:33.522806 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512399    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:51:33.522806 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.512403    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:51:33.522806 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.512958    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 15:51:33.522806 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.521924    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 15:51:33.522806 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.521940    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 15:51:33.522806 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.521961    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 15:51:33.522806 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.521971    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 15:51:33.522806 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522038    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:51:33.522806 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522045    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:51:33.522806 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522048    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:51:33.522806 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522051    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:51:33.522806 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522055    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:51:33.522806 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522058    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:51:33.522806 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522061    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522064    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522067    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522070    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522072    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522075    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522078    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522081    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522084    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522087    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522090    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522092    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522095    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522098    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522100    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522103    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522105    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522108    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522111    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522114    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:51:33.523190 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522117    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522119    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522122    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522126    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522129    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522132    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522134    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522137    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522140    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522142    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522145    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522148    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522150    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522153    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522155    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522158    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522160    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522163    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522166    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522168    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:51:33.523772 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522171    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522173    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522176    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522178    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522181    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522184    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522187    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522189    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522192    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522196    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522199    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522202    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522205    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522208    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522211    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522214    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522217    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522219    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522222    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:51:33.524260 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522225    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522228    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522231    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522233    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522235    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522238    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522241    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522243    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522246    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522249    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522252    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522254    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522257    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522259    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522263    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522266    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522269    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522272    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522274    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522277    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:51:33.524754 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522280    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:51:33.525241 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522282    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:51:33.525241 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522285    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:51:33.525241 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522288    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:51:33.525241 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522291    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:51:33.525241 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522294    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:51:33.525241 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522296    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:51:33.525241 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522299    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:51:33.525241 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522303    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:51:33.525241 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522306    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:51:33.525241 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522308    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:51:33.525241 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522313    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:51:33.525241 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522316    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:51:33.525241 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522319    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:51:33.525241 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.522325    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522461    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522466    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522469    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522473    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522477    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522480    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522483    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522486    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522489    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522492    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522494    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522497    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522500    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522503    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522506    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522509    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522512    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522514    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522517    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:51:33.525602 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522519    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522522    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522525    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522527    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522530    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522533    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522536    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522539    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522541    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522544    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522547    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522549    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522552    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522554    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522556    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522559    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522561    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522565    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522569    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:51:33.526090 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522572    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522574    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522577    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522579    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522582    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522584    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522604    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522609    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522612    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522614    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522617    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522620    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522623    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522625    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522628    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522631    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522633    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522636    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522638    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522642    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:51:33.526554 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522645    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522648    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522650    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522653    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522656    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522658    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522661    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522664    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522667    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522669    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522672    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522674    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522677    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522679    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522682    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522684    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522687    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522689    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522692    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522694    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:51:33.527050 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522698    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522700    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522703    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522705    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522708    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522710    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522713    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522716    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522718    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522720    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522723    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522725    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522728    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522730    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522733    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522735    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522738    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522740    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522743    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522745    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:51:33.527568 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:33.522748    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:51:33.528066 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.522753    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 15:51:33.528066 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.522879    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 15:51:33.528066 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.525923    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 15:51:33.528165 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.528081    2556 server.go:1019] "Starting client certificate rotation"
Sep 09 15:51:33.528210 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.528191    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 15:51:33.528255 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.528243    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 15:51:33.530886 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.530869    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 15:51:33.532029 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.532007    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 15:51:33.553399 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.553375    2556 log.go:25] "Validated CRI v1 runtime API"
Sep 09 15:51:33.555900 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.555879    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 15:51:33.556642 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.556628    2556 log.go:25] "Validated CRI v1 image API"
Sep 09 15:51:33.557526 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.557507    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 15:51:33.560679 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.560661    2556 fs.go:135] Filesystem UUIDs: map[682e59b7-b150-4230-bc56-e9479ee5cadd:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d248518e-0e18-41d0-8996-0e990df9b155:/dev/nvme0n1p4]
Sep 09 15:51:33.560723 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.560678    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 15:51:33.566457 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.566336    2556 manager.go:216] Machine: {Timestamp:2026-09-09 15:51:33.564821795 +0000 UTC m=+0.387207215 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3072493 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec275cd28b51a7fb6f9f74ced9c7bd2d SystemUUID:ec275cd2-8b51-a7fb-6f9f-74ced9c7bd2d BootID:7e0d492f-b862-4b59-906b-37ac37840624 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:86:57:8e:3b:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:86:57:8e:3b:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:e1:6c:71:d2:52 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 15:51:33.566457 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.566452    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 15:51:33.566557 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.566536    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 15:51:33.569649 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.569621    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 15:51:33.569806 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.569651    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-93.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 15:51:33.570512 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.570503    2556 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 15:51:33.570541 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.570515    2556 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 15:51:33.570541 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.570523    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 15:51:33.571186 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.571173    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 15:51:33.572687 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.572677    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 15:51:33.576072 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.576062    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 15:51:33.576213 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.576205    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 15:51:33.576247 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.576239    2556 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 15:51:33.576275 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.576251    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 15:51:33.576275 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.576265    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 15:51:33.576275 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.576274    2556 kubelet.go:398] "Adding apiserver pod source"
Sep 09 15:51:33.576364 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.576284    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 15:51:33.576882 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.576860    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 15:51:33.576964 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.576917    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 15:51:33.577036 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.577019    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-thnps"
Sep 09 15:51:33.580757 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.580711    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 15:51:33.583236 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.583221    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 15:51:33.583300 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.583253    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 15:51:33.583433 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.583420    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 15:51:33.583469 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.583449    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 15:51:33.583469 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.583460    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 15:51:33.583530 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.583470    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 15:51:33.583530 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.583486    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 15:51:33.583530 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.583496    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 15:51:33.583888 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.583878    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 15:51:33.583918 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.583889    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 15:51:33.583918 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.583896    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 15:51:33.583918 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.583901    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 15:51:33.583918 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.583909    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 15:51:33.583918 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.583917    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 15:51:33.584297 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.584283    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-thnps"
Sep 09 15:51:33.584768 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.584758    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 15:51:33.584808 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.584771    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 15:51:33.585741 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:33.585708    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 15:51:33.585863 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:33.585844    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-93.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 15:51:33.588677 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.588662    2556 server.go:1268] "Started kubelet"
Sep 09 15:51:33.588776 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.588724    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 15:51:33.588841 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.588752    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 15:51:33.588841 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.588803    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 15:51:33.588928 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.588875    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-93.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 15:51:33.589603 ip-10-0-133-93 systemd[1]: Started Kubernetes Kubelet.
Sep 09 15:51:33.590128 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.590114    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 15:51:33.591734 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.591720    2556 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 15:51:33.594631 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.594613    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 15:51:33.594726 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:33.594688    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 15:51:33.595487 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.595468    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 15:51:33.595703 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:33.595686    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found"
Sep 09 15:51:33.596225 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.596210    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 15:51:33.596225 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.596225    2556 factory.go:55] Registering systemd factory
Sep 09 15:51:33.596342 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.596231    2556 factory.go:223] Registration of the systemd container factory successfully
Sep 09 15:51:33.596342 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.596285    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 15:51:33.596342 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.596285    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 15:51:33.596342 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.596303    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 15:51:33.596492 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.596430    2556 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 15:51:33.596492 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.596441    2556 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 15:51:33.596492 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.596479    2556 factory.go:153] Registering CRI-O factory
Sep 09 15:51:33.596613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.596495    2556 factory.go:223] Registration of the crio container factory successfully
Sep 09 15:51:33.596613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.596512    2556 factory.go:103] Registering Raw factory
Sep 09 15:51:33.596613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.596527    2556 manager.go:1196] Started watching for new ooms in manager
Sep 09 15:51:33.597422 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.597402    2556 manager.go:349] Starting recovery of all containers
Sep 09 15:51:33.597691 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.597672    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:51:33.600801 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:33.600781    2556 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-93.ec2.internal\" not found" node="ip-10-0-133-93.ec2.internal"
Sep 09 15:51:33.614084 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.613914    2556 manager.go:354] Recovery completed
Sep 09 15:51:33.619632 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.619582    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:51:33.621255 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.621240    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:51:33.621333 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.621271    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:51:33.621333 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.621288    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:51:33.621838 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.621814    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 15:51:33.621838 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.621827    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 15:51:33.621838 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.621830    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:51:33.621838 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.621843    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 15:51:33.622507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.622484    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:51:33.622507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.622509    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:51:33.622624 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.622522    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:51:33.623756 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.623744    2556 policy_none.go:49] "None policy: Start"
Sep 09 15:51:33.623807 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.623759    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 15:51:33.623807 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.623768    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 15:51:33.625414 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.625404    2556 policy_none.go:47] "Start"
Sep 09 15:51:33.756023 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.667208    2556 manager.go:337] "Starting Device Plugin manager"
Sep 09 15:51:33.756023 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:33.667250    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 15:51:33.756023 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.667262    2556 server.go:87] "Starting device plugin registration server"
Sep 09 15:51:33.756023 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.667657    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 15:51:33.756023 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.667670    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 15:51:33.756023 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.667750    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 15:51:33.756023 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.667832    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 15:51:33.756023 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.667841    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 15:51:33.756023 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:33.668461    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 15:51:33.756023 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:33.668508    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-93.ec2.internal\" not found"
Sep 09 15:51:33.756023 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.684833    2556 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2591 into the system container \"/system.slice\": <nil>"
Sep 09 15:51:33.768430 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.768404    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:51:33.769414 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.769391    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:51:33.769518 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.769423    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:51:33.769518 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.769434    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:51:33.769518 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.769462    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-93.ec2.internal"
Sep 09 15:51:33.778821 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.778805    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-93.ec2.internal"
Sep 09 15:51:33.778877 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:33.778826    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-93.ec2.internal\": node \"ip-10-0-133-93.ec2.internal\" not found"
Sep 09 15:51:33.791487 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:33.791462    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found"
Sep 09 15:51:33.799778 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.799756    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 15:51:33.800979 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.800960    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 15:51:33.801060 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.801006    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 15:51:33.801060 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.801027    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 15:51:33.801202 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:33.801127    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 15:51:33.803105 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.803084    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:51:33.892021 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:33.892000    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found"
Sep 09 15:51:33.901442 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.901422    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal"]
Sep 09 15:51:33.901497 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.901488    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:51:33.902945 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.902930    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:51:33.903034 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.902962    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:51:33.903034 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.902976    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:51:33.905258 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.905243    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:51:33.905371 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.905357    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal"
Sep 09 15:51:33.905416 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.905385    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:51:33.905961 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.905946    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:51:33.906048 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.905976    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:51:33.906048 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.905990    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:51:33.906048 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.905948    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:51:33.906187 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.906055    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:51:33.906187 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.906065    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:51:33.908204 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.908181    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal"
Sep 09 15:51:33.908286 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.908220    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:51:33.908915 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.908899    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:51:33.908980 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.908928    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:51:33.908980 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.908938    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:51:33.925113 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:33.925095    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-93.ec2.internal\" not found" node="ip-10-0-133-93.ec2.internal"
Sep 09 15:51:33.928991 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:33.928974    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-93.ec2.internal\" not found" node="ip-10-0-133-93.ec2.internal"
Sep 09 15:51:33.992298 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:33.992268    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found"
Sep 09 15:51:33.998675 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.998656    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2ab3a812dc27588fde74d62d7cf30f3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-93.ec2.internal\" (UID: \"a2ab3a812dc27588fde74d62d7cf30f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal"
Sep 09 15:51:33.998737 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.998681    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2859ad4c0a6499bd80e58f83465fc454-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal\" (UID: \"2859ad4c0a6499bd80e58f83465fc454\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal"
Sep 09 15:51:33.998737 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:33.998709    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2859ad4c0a6499bd80e58f83465fc454-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal\" (UID: \"2859ad4c0a6499bd80e58f83465fc454\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal"
Sep 09 15:51:34.093077 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:34.093016    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found"
Sep 09 15:51:34.099404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:34.099382    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2ab3a812dc27588fde74d62d7cf30f3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-93.ec2.internal\" (UID: \"a2ab3a812dc27588fde74d62d7cf30f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal"
Sep 09 15:51:34.099480 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:34.099413    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2859ad4c0a6499bd80e58f83465fc454-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal\" (UID: \"2859ad4c0a6499bd80e58f83465fc454\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal"
Sep 09 15:51:34.099480 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:34.099430    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2859ad4c0a6499bd80e58f83465fc454-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal\" (UID: \"2859ad4c0a6499bd80e58f83465fc454\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal"
Sep 09 15:51:34.099480 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:34.099467    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2859ad4c0a6499bd80e58f83465fc454-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal\" (UID: \"2859ad4c0a6499bd80e58f83465fc454\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal"
Sep 09 15:51:34.099480 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:34.099473    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2ab3a812dc27588fde74d62d7cf30f3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-93.ec2.internal\" (UID: \"a2ab3a812dc27588fde74d62d7cf30f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal"
Sep 09 15:51:34.099639 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:34.099487    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2859ad4c0a6499bd80e58f83465fc454-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal\" (UID: \"2859ad4c0a6499bd80e58f83465fc454\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal"
Sep 09 15:51:36.149479 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:36.149427    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found"
Sep 09 15:51:36.183938 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:36.183913    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal"
Sep 09 15:51:36.187325 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:36.187300    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal"
Sep 09 15:51:36.249992 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:36.249960    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found"
Sep 09 15:51:36.350559 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:36.350496    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found"
Sep 09 15:51:36.450993 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:36.450965    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found"
Sep 09 15:51:36.483409 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:36.483388    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 15:51:36.483854 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:36.483562    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 15:51:36.483854 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:36.483599    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 15:51:36.545000 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:36.544948    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 15:46:35 +0000 UTC" deadline="2028-05-21 14:54:56.906135292 +0000 UTC"
Sep 09 15:51:36.545000 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:36.544990    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14879h3m20.361148644s"
Sep 09 15:51:36.551209 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:36.551178    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-93.ec2.internal\" not found"
Sep 09 15:51:36.609856 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:36.609770    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:51:36.652034 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:36.651998    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal"
Sep 09 15:51:36.662759 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:36.662735    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 15:51:36.664903 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:36.664890    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal"
Sep 09 15:51:36.670587 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:36.670572    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 15:51:36.949594 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:36.949525    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:51:36.967584 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:36.967555    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ab3a812dc27588fde74d62d7cf30f3.slice/crio-a6448898c820b86860c13245e42fd8d849b67c0f81f0af74c9855d5e233ac202 WatchSource:0}: Error finding container a6448898c820b86860c13245e42fd8d849b67c0f81f0af74c9855d5e233ac202: Status 404 returned error can't find the container with id a6448898c820b86860c13245e42fd8d849b67c0f81f0af74c9855d5e233ac202
Sep 09 15:51:36.968089 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:36.968066    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2859ad4c0a6499bd80e58f83465fc454.slice/crio-37b20e341afef26fe488e8d4ca35d917ba44f68913029d4391fe9d9dd6a3156f WatchSource:0}: Error finding container 37b20e341afef26fe488e8d4ca35d917ba44f68913029d4391fe9d9dd6a3156f: Status 404 returned error can't find the container with id 37b20e341afef26fe488e8d4ca35d917ba44f68913029d4391fe9d9dd6a3156f
Sep 09 15:51:36.971493 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:36.971449    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 15:51:37.498944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.498909    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:51:37.533783 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.533748    2556 apiserver.go:52] "Watching apiserver"
Sep 09 15:51:37.539754 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.539730    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 15:51:37.542301 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.542271    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-ms57r","kube-system/global-pull-secret-syncer-7fdsm","kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj","openshift-dns/node-resolver-dzktt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal","openshift-multus/multus-additional-cni-plugins-xp9wv","openshift-multus/multus-lxvn6","openshift-multus/network-metrics-daemon-4tmj4","kube-system/konnectivity-agent-5jrvg","openshift-cluster-node-tuning-operator/tuned-wv8m7","openshift-image-registry/node-ca-kx87f","openshift-network-diagnostics/network-check-target-xsht8","openshift-network-operator/iptables-alerter-f47qf"]
Sep 09 15:51:37.544856 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.544833    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.547036 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.547015    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:37.547119 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:37.547094    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:51:37.547182 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.547164    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 15:51:37.547420 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.547399    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 15:51:37.547533 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.547448    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 15:51:37.547653 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.547633    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 15:51:37.547653 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.547649    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6hdm9\""
Sep 09 15:51:37.547809 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.547715    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 15:51:37.549264 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.549242    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f47qf"
Sep 09 15:51:37.551353 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.551335    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 15:51:37.551760 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.551737    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 15:51:37.551860 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.551792    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2qjpd\""
Sep 09 15:51:37.551922 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.551859    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 15:51:37.551922 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.551888    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 15:51:37.552019 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.551988    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.553924 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.553895    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 15:51:37.554026 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.554015    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dzktt"
Sep 09 15:51:37.554136 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.554114    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 15:51:37.554420 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.554401    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 15:51:37.554522 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.554403    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dsnwk\""
Sep 09 15:51:37.555985 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.555966    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 15:51:37.556079 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.556070    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 15:51:37.556173 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.556090    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ssmlc\""
Sep 09 15:51:37.556327 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.556310    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.558270 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.558250    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 15:51:37.558576 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.558558    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xlh9x\""
Sep 09 15:51:37.558666 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.558598    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:37.558720 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:37.558664    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:51:37.560596 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560444    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/322726ce-594b-4b74-968b-40ad101cb1fb-system-cni-dir\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.560596 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560506    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8627dab-9db2-4cb5-9ee9-37c644998bbb-iptables-alerter-script\") pod \"iptables-alerter-f47qf\" (UID: \"d8627dab-9db2-4cb5-9ee9-37c644998bbb\") " pod="openshift-network-operator/iptables-alerter-f47qf"
Sep 09 15:51:37.560596 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560541    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2kpx\" (UniqueName: \"kubernetes.io/projected/d8627dab-9db2-4cb5-9ee9-37c644998bbb-kube-api-access-q2kpx\") pod \"iptables-alerter-f47qf\" (UID: \"d8627dab-9db2-4cb5-9ee9-37c644998bbb\") " pod="openshift-network-operator/iptables-alerter-f47qf"
Sep 09 15:51:37.560755 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560590    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-registration-dir\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.560755 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560622    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dffd2\" (UniqueName: \"kubernetes.io/projected/0a678faa-f9a8-4199-b722-b371b9dc2b59-kube-api-access-dffd2\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.560755 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560650    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d08d290-7a84-4502-8534-3c03383e94f0-hosts-file\") pod \"node-resolver-dzktt\" (UID: \"8d08d290-7a84-4502-8534-3c03383e94f0\") " pod="openshift-dns/node-resolver-dzktt"
Sep 09 15:51:37.560755 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560674    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/322726ce-594b-4b74-968b-40ad101cb1fb-cnibin\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.560755 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560713    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/322726ce-594b-4b74-968b-40ad101cb1fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.560755 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560738    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/322726ce-594b-4b74-968b-40ad101cb1fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.561038 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560763    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-device-dir\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.561038 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560787    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-sys-fs\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.561038 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560828    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d08d290-7a84-4502-8534-3c03383e94f0-tmp-dir\") pod \"node-resolver-dzktt\" (UID: \"8d08d290-7a84-4502-8534-3c03383e94f0\") " pod="openshift-dns/node-resolver-dzktt"
Sep 09 15:51:37.561038 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560868    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/322726ce-594b-4b74-968b-40ad101cb1fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.561038 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560907    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/322726ce-594b-4b74-968b-40ad101cb1fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.561038 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560934    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpk84\" (UniqueName: \"kubernetes.io/projected/322726ce-594b-4b74-968b-40ad101cb1fb-kube-api-access-fpk84\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.561038 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.560967    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8627dab-9db2-4cb5-9ee9-37c644998bbb-host-slash\") pod \"iptables-alerter-f47qf\" (UID: \"d8627dab-9db2-4cb5-9ee9-37c644998bbb\") " pod="openshift-network-operator/iptables-alerter-f47qf"
Sep 09 15:51:37.561038 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.561009    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-socket-dir\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.561410 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.561058    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gsz4\" (UniqueName: \"kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4\") pod \"network-check-target-xsht8\" (UID: \"7bafb10d-a48d-4429-b1a4-532d3ed5a475\") " pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:37.561410 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.561082    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.561410 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.561121    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-etc-selinux\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.561410 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.561145    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcs9l\" (UniqueName: \"kubernetes.io/projected/8d08d290-7a84-4502-8534-3c03383e94f0-kube-api-access-qcs9l\") pod \"node-resolver-dzktt\" (UID: \"8d08d290-7a84-4502-8534-3c03383e94f0\") " pod="openshift-dns/node-resolver-dzktt"
Sep 09 15:51:37.562137 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.562115    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 15:51:37.562215 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.562134    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.564216 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.564197    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 15:51:37.564361 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.564343    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 15:51:37.564476 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.564438    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 15:51:37.564650 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.564590    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 15:51:37.564650 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.564601    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9pdd7\""
Sep 09 15:51:37.565057 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.564592    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 15:51:37.565057 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.564883    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 15:51:37.565057 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.564936    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:37.565057 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:37.565000    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:51:37.569192 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.569157    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5jrvg"
Sep 09 15:51:37.571963 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.571945    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 15:51:37.572055 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.572038    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dnnjn\""
Sep 09 15:51:37.572112 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.571951    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 15:51:37.572440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.572423    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.574616 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.574594    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 15:51:37.574710 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.574683    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rs6q7\""
Sep 09 15:51:37.574893 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.574875    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kx87f"
Sep 09 15:51:37.574972 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.574915    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 15:51:37.576795 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.576777    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 15:51:37.577069 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.577057    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 15:51:37.577142 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.577124    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 15:51:37.577668 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.577650    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x2swg\""
Sep 09 15:51:37.584675 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.584656    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-97mnh"
Sep 09 15:51:37.594511 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.594492    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-97mnh"
Sep 09 15:51:37.653634 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.653607    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 15:51:37.661703 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.661675    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgkv8\" (UniqueName: \"kubernetes.io/projected/72246d50-b16d-4f50-b064-aa2a85cf2970-kube-api-access-zgkv8\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:37.661829 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.661717    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-run-openvswitch\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.661829 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.661744    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lcgf\" (UniqueName: \"kubernetes.io/projected/80563802-ee38-4620-addd-926a31290be2-kube-api-access-2lcgf\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.661829 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.661768    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpkkm\" (UniqueName: \"kubernetes.io/projected/c1c0dea1-1500-498d-ade5-e8bd61bbcd0b-kube-api-access-qpkkm\") pod \"node-ca-kx87f\" (UID: \"c1c0dea1-1500-498d-ade5-e8bd61bbcd0b\") " pod="openshift-image-registry/node-ca-kx87f"
Sep 09 15:51:37.661829 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.661791    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-run-multus-certs\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.661829 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.661821    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/322726ce-594b-4b74-968b-40ad101cb1fb-cnibin\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.662085 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.661847    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/322726ce-594b-4b74-968b-40ad101cb1fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.662085 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.661870    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-etc-openvswitch\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.662085 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.661908    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-run-ovn-kubernetes\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.662085 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.661932    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-run-netns\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.662085 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.661956    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/322726ce-594b-4b74-968b-40ad101cb1fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.662085 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.661990    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-kubelet\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.662085 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662012    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80563802-ee38-4620-addd-926a31290be2-ovnkube-script-lib\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.662085 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662035    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-run-k8s-cni-cncf-io\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.662085 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662058    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-run-systemd\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.662085 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662081    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-cni-bin\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662102    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-kubernetes\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662155    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1c0dea1-1500-498d-ade5-e8bd61bbcd0b-host\") pod \"node-ca-kx87f\" (UID: \"c1c0dea1-1500-498d-ade5-e8bd61bbcd0b\") " pod="openshift-image-registry/node-ca-kx87f"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662179    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-multus-cni-dir\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662203    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-multus-socket-dir-parent\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662230    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-hostroot\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662254    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d08d290-7a84-4502-8534-3c03383e94f0-tmp-dir\") pod \"node-resolver-dzktt\" (UID: \"8d08d290-7a84-4502-8534-3c03383e94f0\") " pod="openshift-dns/node-resolver-dzktt"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662281    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-node-log\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662317    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-log-socket\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662339    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-sys\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662361    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-cnibin\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662383    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-var-lib-cni-bin\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662407    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/322726ce-594b-4b74-968b-40ad101cb1fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662432    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-socket-dir\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662458    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-sysctl-d\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662497    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-host\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662518    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc5d1009-a15f-4c28-b291-074695366f79-cni-binary-copy\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.662613 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662541    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-multus-conf-dir\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662564    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c5f3c15-9eee-43d1-b403-f89806e864cb-dbus\") pod \"global-pull-secret-syncer-7fdsm\" (UID: \"9c5f3c15-9eee-43d1-b403-f89806e864cb\") " pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662593    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-etc-selinux\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662616    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-run-netns\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662642    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8627dab-9db2-4cb5-9ee9-37c644998bbb-iptables-alerter-script\") pod \"iptables-alerter-f47qf\" (UID: \"d8627dab-9db2-4cb5-9ee9-37c644998bbb\") " pod="openshift-network-operator/iptables-alerter-f47qf"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662665    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80563802-ee38-4620-addd-926a31290be2-ovn-node-metrics-cert\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662692    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-device-dir\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662792    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-sys-fs\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662830    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-run-ovn\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662857    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80563802-ee38-4620-addd-926a31290be2-env-overrides\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662891    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-modprobe-d\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662898    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-socket-dir\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662920    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/322726ce-594b-4b74-968b-40ad101cb1fb-cnibin\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662926    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-device-dir\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662930    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-sysconfig\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662973    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c5f3c15-9eee-43d1-b403-f89806e864cb-kubelet-config\") pod \"global-pull-secret-syncer-7fdsm\" (UID: \"9c5f3c15-9eee-43d1-b403-f89806e864cb\") " pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:37.663318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.662987    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-sys-fs\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663000    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret\") pod \"global-pull-secret-syncer-7fdsm\" (UID: \"9c5f3c15-9eee-43d1-b403-f89806e864cb\") " pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663033    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8627dab-9db2-4cb5-9ee9-37c644998bbb-host-slash\") pod \"iptables-alerter-f47qf\" (UID: \"d8627dab-9db2-4cb5-9ee9-37c644998bbb\") " pod="openshift-network-operator/iptables-alerter-f47qf"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663040    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-etc-selinux\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663057    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663087    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8627dab-9db2-4cb5-9ee9-37c644998bbb-host-slash\") pod \"iptables-alerter-f47qf\" (UID: \"d8627dab-9db2-4cb5-9ee9-37c644998bbb\") " pod="openshift-network-operator/iptables-alerter-f47qf"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663120    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-run\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663169    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcs9l\" (UniqueName: \"kubernetes.io/projected/8d08d290-7a84-4502-8534-3c03383e94f0-kube-api-access-qcs9l\") pod \"node-resolver-dzktt\" (UID: \"8d08d290-7a84-4502-8534-3c03383e94f0\") " pod="openshift-dns/node-resolver-dzktt"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663200    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663233    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-slash\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663334    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-var-lib-kubelet\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663361    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-etc-kubernetes\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663374    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8627dab-9db2-4cb5-9ee9-37c644998bbb-iptables-alerter-script\") pod \"iptables-alerter-f47qf\" (UID: \"d8627dab-9db2-4cb5-9ee9-37c644998bbb\") " pod="openshift-network-operator/iptables-alerter-f47qf"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663406    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/322726ce-594b-4b74-968b-40ad101cb1fb-system-cni-dir\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663446    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-registration-dir\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663507    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d08d290-7a84-4502-8534-3c03383e94f0-hosts-file\") pod \"node-resolver-dzktt\" (UID: \"8d08d290-7a84-4502-8534-3c03383e94f0\") " pod="openshift-dns/node-resolver-dzktt"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663509    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/322726ce-594b-4b74-968b-40ad101cb1fb-system-cni-dir\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.664109 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663531    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-systemd-units\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663541    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-registration-dir\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663558    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d08d290-7a84-4502-8534-3c03383e94f0-hosts-file\") pod \"node-resolver-dzktt\" (UID: \"8d08d290-7a84-4502-8534-3c03383e94f0\") " pod="openshift-dns/node-resolver-dzktt"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663562    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/322726ce-594b-4b74-968b-40ad101cb1fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663588    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80563802-ee38-4620-addd-926a31290be2-ovnkube-config\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663605    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-lib-modules\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663642    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-tmp\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663672    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq6x9\" (UniqueName: \"kubernetes.io/projected/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-kube-api-access-jq6x9\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663707    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d08d290-7a84-4502-8534-3c03383e94f0-tmp-dir\") pod \"node-resolver-dzktt\" (UID: \"8d08d290-7a84-4502-8534-3c03383e94f0\") " pod="openshift-dns/node-resolver-dzktt"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.663774    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/322726ce-594b-4b74-968b-40ad101cb1fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.664021    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/322726ce-594b-4b74-968b-40ad101cb1fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.664131    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/322726ce-594b-4b74-968b-40ad101cb1fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.664193    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-var-lib-openvswitch\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.664228    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-sysctl-conf\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.664286    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-var-lib-cni-multus\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.664311    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0880616b-7f54-4880-816e-f21755439fdf-agent-certs\") pod \"konnectivity-agent-5jrvg\" (UID: \"0880616b-7f54-4880-816e-f21755439fdf\") " pod="kube-system/konnectivity-agent-5jrvg"
Sep 09 15:51:37.664933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.664333    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0880616b-7f54-4880-816e-f21755439fdf-konnectivity-ca\") pod \"konnectivity-agent-5jrvg\" (UID: \"0880616b-7f54-4880-816e-f21755439fdf\") " pod="kube-system/konnectivity-agent-5jrvg"
Sep 09 15:51:37.665729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.664755    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/322726ce-594b-4b74-968b-40ad101cb1fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.665729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.664826    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpk84\" (UniqueName: \"kubernetes.io/projected/322726ce-594b-4b74-968b-40ad101cb1fb-kube-api-access-fpk84\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.665729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.664882    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-systemd\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.665729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.664919    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-var-lib-kubelet\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.665729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.664949    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-system-cni-dir\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.665729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.664978    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gsz4\" (UniqueName: \"kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4\") pod \"network-check-target-xsht8\" (UID: \"7bafb10d-a48d-4429-b1a4-532d3ed5a475\") " pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:37.665729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.665005    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.665729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.665055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-cni-netd\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.665729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.665082    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-tuned\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.665729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.665109    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1c0dea1-1500-498d-ade5-e8bd61bbcd0b-serviceca\") pod \"node-ca-kx87f\" (UID: \"c1c0dea1-1500-498d-ade5-e8bd61bbcd0b\") " pod="openshift-image-registry/node-ca-kx87f"
Sep 09 15:51:37.665729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.665131    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc5d1009-a15f-4c28-b291-074695366f79-multus-daemon-config\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.665729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.665151    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a678faa-f9a8-4199-b722-b371b9dc2b59-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.665729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.665155    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjpnc\" (UniqueName: \"kubernetes.io/projected/cc5d1009-a15f-4c28-b291-074695366f79-kube-api-access-xjpnc\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.665729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.665213    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2kpx\" (UniqueName: \"kubernetes.io/projected/d8627dab-9db2-4cb5-9ee9-37c644998bbb-kube-api-access-q2kpx\") pod \"iptables-alerter-f47qf\" (UID: \"d8627dab-9db2-4cb5-9ee9-37c644998bbb\") " pod="openshift-network-operator/iptables-alerter-f47qf"
Sep 09 15:51:37.665729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.665239    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dffd2\" (UniqueName: \"kubernetes.io/projected/0a678faa-f9a8-4199-b722-b371b9dc2b59-kube-api-access-dffd2\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.668958 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.668933    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 15:51:37.670609 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:37.670589    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:51:37.670714 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:37.670615    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:51:37.670714 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:37.670628    2556 projected.go:196] Error preparing data for projected volume kube-api-access-7gsz4 for pod openshift-network-diagnostics/network-check-target-xsht8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:51:37.670714 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:37.670692    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4 podName:7bafb10d-a48d-4429-b1a4-532d3ed5a475 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:38.170666365 +0000 UTC m=+3.037177576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7gsz4" (UniqueName: "kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4") pod "network-check-target-xsht8" (UID: "7bafb10d-a48d-4429-b1a4-532d3ed5a475") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:51:37.675384 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.675361    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dffd2\" (UniqueName: \"kubernetes.io/projected/0a678faa-f9a8-4199-b722-b371b9dc2b59-kube-api-access-dffd2\") pod \"aws-ebs-csi-driver-node-6sffj\" (UID: \"0a678faa-f9a8-4199-b722-b371b9dc2b59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.675501 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.675368    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcs9l\" (UniqueName: \"kubernetes.io/projected/8d08d290-7a84-4502-8534-3c03383e94f0-kube-api-access-qcs9l\") pod \"node-resolver-dzktt\" (UID: \"8d08d290-7a84-4502-8534-3c03383e94f0\") " pod="openshift-dns/node-resolver-dzktt"
Sep 09 15:51:37.675797 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.675773    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpk84\" (UniqueName: \"kubernetes.io/projected/322726ce-594b-4b74-968b-40ad101cb1fb-kube-api-access-fpk84\") pod \"multus-additional-cni-plugins-xp9wv\" (UID: \"322726ce-594b-4b74-968b-40ad101cb1fb\") " pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.676984 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.676381    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2kpx\" (UniqueName: \"kubernetes.io/projected/d8627dab-9db2-4cb5-9ee9-37c644998bbb-kube-api-access-q2kpx\") pod \"iptables-alerter-f47qf\" (UID: \"d8627dab-9db2-4cb5-9ee9-37c644998bbb\") " pod="openshift-network-operator/iptables-alerter-f47qf"
Sep 09 15:51:37.763358 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.763247    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" event={"ID":"2859ad4c0a6499bd80e58f83465fc454","Type":"ContainerStarted","Data":"37b20e341afef26fe488e8d4ca35d917ba44f68913029d4391fe9d9dd6a3156f"}
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765523    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret\") pod \"global-pull-secret-syncer-7fdsm\" (UID: \"9c5f3c15-9eee-43d1-b403-f89806e864cb\") " pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765564    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765590    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-run\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765617    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765620    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal" event={"ID":"a2ab3a812dc27588fde74d62d7cf30f3","Type":"ContainerStarted","Data":"a6448898c820b86860c13245e42fd8d849b67c0f81f0af74c9855d5e233ac202"}
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765639    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-slash\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765660    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-var-lib-kubelet\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765682    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-etc-kubernetes\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765696    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765707    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-systemd-units\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765728    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80563802-ee38-4620-addd-926a31290be2-ovnkube-config\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765749    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-lib-modules\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765770    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-tmp\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765789    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq6x9\" (UniqueName: \"kubernetes.io/projected/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-kube-api-access-jq6x9\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:37.765793    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765814    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-var-lib-openvswitch\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.767944 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765834    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-sysctl-conf\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:37.765853    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret podName:9c5f3c15-9eee-43d1-b403-f89806e864cb nodeName:}" failed. No retries permitted until 2026-09-09 15:51:38.265833956 +0000 UTC m=+3.132345176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret") pod "global-pull-secret-syncer-7fdsm" (UID: "9c5f3c15-9eee-43d1-b403-f89806e864cb") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765877    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-var-lib-cni-multus\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765898    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0880616b-7f54-4880-816e-f21755439fdf-agent-certs\") pod \"konnectivity-agent-5jrvg\" (UID: \"0880616b-7f54-4880-816e-f21755439fdf\") " pod="kube-system/konnectivity-agent-5jrvg"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765919    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0880616b-7f54-4880-816e-f21755439fdf-konnectivity-ca\") pod \"konnectivity-agent-5jrvg\" (UID: \"0880616b-7f54-4880-816e-f21755439fdf\") " pod="kube-system/konnectivity-agent-5jrvg"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765941    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-systemd\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765965    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-sysctl-conf\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.765967    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-var-lib-kubelet\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766004    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-system-cni-dir\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766041    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-cni-netd\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766070    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-tuned\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766104    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1c0dea1-1500-498d-ade5-e8bd61bbcd0b-serviceca\") pod \"node-ca-kx87f\" (UID: \"c1c0dea1-1500-498d-ade5-e8bd61bbcd0b\") " pod="openshift-image-registry/node-ca-kx87f"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766128    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc5d1009-a15f-4c28-b291-074695366f79-multus-daemon-config\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766151    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjpnc\" (UniqueName: \"kubernetes.io/projected/cc5d1009-a15f-4c28-b291-074695366f79-kube-api-access-xjpnc\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766179    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgkv8\" (UniqueName: \"kubernetes.io/projected/72246d50-b16d-4f50-b064-aa2a85cf2970-kube-api-access-zgkv8\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766195    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-var-lib-cni-multus\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766203    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-run-openvswitch\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.768861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766226    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lcgf\" (UniqueName: \"kubernetes.io/projected/80563802-ee38-4620-addd-926a31290be2-kube-api-access-2lcgf\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766248    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpkkm\" (UniqueName: \"kubernetes.io/projected/c1c0dea1-1500-498d-ade5-e8bd61bbcd0b-kube-api-access-qpkkm\") pod \"node-ca-kx87f\" (UID: \"c1c0dea1-1500-498d-ade5-e8bd61bbcd0b\") " pod="openshift-image-registry/node-ca-kx87f"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766272    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-run-multus-certs\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766297    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-etc-openvswitch\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766311    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-var-lib-openvswitch\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766323    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-run-ovn-kubernetes\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766346    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-run-netns\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766350    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-etc-kubernetes\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766353    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-systemd\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766376    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-kubelet\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766397    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-slash\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766420    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80563802-ee38-4620-addd-926a31290be2-ovnkube-script-lib\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766445    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-run\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766446    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-run-k8s-cni-cncf-io\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766503    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-run-k8s-cni-cncf-io\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766507    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-run-systemd\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766537    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-cni-bin\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766560    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-kubernetes\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.769731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766574    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1c0dea1-1500-498d-ade5-e8bd61bbcd0b-host\") pod \"node-ca-kx87f\" (UID: \"c1c0dea1-1500-498d-ade5-e8bd61bbcd0b\") " pod="openshift-image-registry/node-ca-kx87f"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766613    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-multus-cni-dir\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766645    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-multus-socket-dir-parent\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766669    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-hostroot\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766692    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-node-log\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766712    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-log-socket\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766735    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-sys\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766755    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-cnibin\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766763    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-run-openvswitch\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766777    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-var-lib-cni-bin\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766800    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-sysctl-d\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766834    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-host\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766856    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc5d1009-a15f-4c28-b291-074695366f79-cni-binary-copy\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766863    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80563802-ee38-4620-addd-926a31290be2-ovnkube-config\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766877    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-multus-conf-dir\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766891    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc5d1009-a15f-4c28-b291-074695366f79-multus-daemon-config\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766902    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c5f3c15-9eee-43d1-b403-f89806e864cb-dbus\") pod \"global-pull-secret-syncer-7fdsm\" (UID: \"9c5f3c15-9eee-43d1-b403-f89806e864cb\") " pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766896    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0880616b-7f54-4880-816e-f21755439fdf-konnectivity-ca\") pod \"konnectivity-agent-5jrvg\" (UID: \"0880616b-7f54-4880-816e-f21755439fdf\") " pod="kube-system/konnectivity-agent-5jrvg"
Sep 09 15:51:37.770573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766928    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-run-netns\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766935    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-cni-netd\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766955    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80563802-ee38-4620-addd-926a31290be2-ovn-node-metrics-cert\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766977    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-etc-openvswitch\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766983    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-run-ovn\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766986    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-lib-modules\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767003    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-systemd-units\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767034    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-run-ovn\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767057    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-run-systemd\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767059    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-kubelet\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766008    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-var-lib-kubelet\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766420    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-var-lib-kubelet\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.766978    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-run-ovn-kubernetes\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767125    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80563802-ee38-4620-addd-926a31290be2-env-overrides\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767153    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-modprobe-d\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767179    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-sysconfig\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767182    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1c0dea1-1500-498d-ade5-e8bd61bbcd0b-host\") pod \"node-ca-kx87f\" (UID: \"c1c0dea1-1500-498d-ade5-e8bd61bbcd0b\") " pod="openshift-image-registry/node-ca-kx87f"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767204    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c5f3c15-9eee-43d1-b403-f89806e864cb-kubelet-config\") pod \"global-pull-secret-syncer-7fdsm\" (UID: \"9c5f3c15-9eee-43d1-b403-f89806e864cb\") " pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:37.771440 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767230    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-kubernetes\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767251    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c5f3c15-9eee-43d1-b403-f89806e864cb-dbus\") pod \"global-pull-secret-syncer-7fdsm\" (UID: \"9c5f3c15-9eee-43d1-b403-f89806e864cb\") " pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767267    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-run-multus-certs\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767300    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-run-netns\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767310    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-multus-socket-dir-parent\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767341    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-hostroot\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767348    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-multus-cni-dir\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767375    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-node-log\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767544    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc5d1009-a15f-4c28-b291-074695366f79-cni-binary-copy\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767579    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-multus-conf-dir\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767630    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-cnibin\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767676    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-sys\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767709    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-host-cni-bin\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767740    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80563802-ee38-4620-addd-926a31290be2-log-socket\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767776    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-var-lib-cni-bin\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767830    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80563802-ee38-4620-addd-926a31290be2-env-overrides\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767908    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80563802-ee38-4620-addd-926a31290be2-ovnkube-script-lib\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767928    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-modprobe-d\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.772243 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767920    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-sysctl-d\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.772965 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767069    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-system-cni-dir\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.772965 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:37.767990    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:51:37.772965 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.768031    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c5f3c15-9eee-43d1-b403-f89806e864cb-kubelet-config\") pod \"global-pull-secret-syncer-7fdsm\" (UID: \"9c5f3c15-9eee-43d1-b403-f89806e864cb\") " pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:37.772965 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.767997    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc5d1009-a15f-4c28-b291-074695366f79-host-run-netns\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.772965 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.768034    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-sysconfig\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.772965 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:37.768062    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs podName:72246d50-b16d-4f50-b064-aa2a85cf2970 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:38.268046043 +0000 UTC m=+3.134557254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs") pod "network-metrics-daemon-4tmj4" (UID: "72246d50-b16d-4f50-b064-aa2a85cf2970") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:51:37.772965 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.768098    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-host\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.772965 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.768153    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1c0dea1-1500-498d-ade5-e8bd61bbcd0b-serviceca\") pod \"node-ca-kx87f\" (UID: \"c1c0dea1-1500-498d-ade5-e8bd61bbcd0b\") " pod="openshift-image-registry/node-ca-kx87f"
Sep 09 15:51:37.772965 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.769940    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80563802-ee38-4620-addd-926a31290be2-ovn-node-metrics-cert\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.772965 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.770010    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-tmp\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.772965 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.771414    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0880616b-7f54-4880-816e-f21755439fdf-agent-certs\") pod \"konnectivity-agent-5jrvg\" (UID: \"0880616b-7f54-4880-816e-f21755439fdf\") " pod="kube-system/konnectivity-agent-5jrvg"
Sep 09 15:51:37.772965 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.771533    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-etc-tuned\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.776179 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.776089    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjpnc\" (UniqueName: \"kubernetes.io/projected/cc5d1009-a15f-4c28-b291-074695366f79-kube-api-access-xjpnc\") pod \"multus-lxvn6\" (UID: \"cc5d1009-a15f-4c28-b291-074695366f79\") " pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.776335 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.776232    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpkkm\" (UniqueName: \"kubernetes.io/projected/c1c0dea1-1500-498d-ade5-e8bd61bbcd0b-kube-api-access-qpkkm\") pod \"node-ca-kx87f\" (UID: \"c1c0dea1-1500-498d-ade5-e8bd61bbcd0b\") " pod="openshift-image-registry/node-ca-kx87f"
Sep 09 15:51:37.777794 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.777626    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgkv8\" (UniqueName: \"kubernetes.io/projected/72246d50-b16d-4f50-b064-aa2a85cf2970-kube-api-access-zgkv8\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:37.777794 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.777754    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lcgf\" (UniqueName: \"kubernetes.io/projected/80563802-ee38-4620-addd-926a31290be2-kube-api-access-2lcgf\") pod \"ovnkube-node-ms57r\" (UID: \"80563802-ee38-4620-addd-926a31290be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.778002 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.777984    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq6x9\" (UniqueName: \"kubernetes.io/projected/63f36a9e-8e7a-49cf-b8f6-635aa633bfa9-kube-api-access-jq6x9\") pod \"tuned-wv8m7\" (UID: \"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9\") " pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.856970 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.856940    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xp9wv"
Sep 09 15:51:37.864759 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.864738    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f47qf"
Sep 09 15:51:37.871066 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:37.871039    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322726ce_594b_4b74_968b_40ad101cb1fb.slice/crio-ff23b3bc2428679a6683eeadc1860470d0dd2cc465afdd25ddca1ebadbb0da33 WatchSource:0}: Error finding container ff23b3bc2428679a6683eeadc1860470d0dd2cc465afdd25ddca1ebadbb0da33: Status 404 returned error can't find the container with id ff23b3bc2428679a6683eeadc1860470d0dd2cc465afdd25ddca1ebadbb0da33
Sep 09 15:51:37.874875 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.874727    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj"
Sep 09 15:51:37.880853 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.880109    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dzktt"
Sep 09 15:51:37.887861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.887843    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lxvn6"
Sep 09 15:51:37.888424 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:37.888398    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a678faa_f9a8_4199_b722_b371b9dc2b59.slice/crio-39075e9b08e88b9aab2f2119bea5fcce42af6809500f2cfc347ba92c912f8897 WatchSource:0}: Error finding container 39075e9b08e88b9aab2f2119bea5fcce42af6809500f2cfc347ba92c912f8897: Status 404 returned error can't find the container with id 39075e9b08e88b9aab2f2119bea5fcce42af6809500f2cfc347ba92c912f8897
Sep 09 15:51:37.890610 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:37.890588    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d08d290_7a84_4502_8534_3c03383e94f0.slice/crio-21e4586aca403f6100a4556eda5cb885e9fd21c723590d0a8c078c2d9ff8c298 WatchSource:0}: Error finding container 21e4586aca403f6100a4556eda5cb885e9fd21c723590d0a8c078c2d9ff8c298: Status 404 returned error can't find the container with id 21e4586aca403f6100a4556eda5cb885e9fd21c723590d0a8c078c2d9ff8c298
Sep 09 15:51:37.894608 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.894588    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:51:37.901352 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.901290    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5jrvg"
Sep 09 15:51:37.903589 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:37.903559    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5d1009_a15f_4c28_b291_074695366f79.slice/crio-1699d9d82c1a3359bdd7176fb005b403f92bb979637d1488262a8a2f77e1eec5 WatchSource:0}: Error finding container 1699d9d82c1a3359bdd7176fb005b403f92bb979637d1488262a8a2f77e1eec5: Status 404 returned error can't find the container with id 1699d9d82c1a3359bdd7176fb005b403f92bb979637d1488262a8a2f77e1eec5
Sep 09 15:51:37.909223 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.909178    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wv8m7"
Sep 09 15:51:37.911078 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:37.910863    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80563802_ee38_4620_addd_926a31290be2.slice/crio-cc154d619fa33580bf5e187fcbf7b141e4a412f7428ce8e5eda98d7694ee87c0 WatchSource:0}: Error finding container cc154d619fa33580bf5e187fcbf7b141e4a412f7428ce8e5eda98d7694ee87c0: Status 404 returned error can't find the container with id cc154d619fa33580bf5e187fcbf7b141e4a412f7428ce8e5eda98d7694ee87c0
Sep 09 15:51:37.916329 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:37.916295    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kx87f"
Sep 09 15:51:37.918096 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:37.918068    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0880616b_7f54_4880_816e_f21755439fdf.slice/crio-d351bff816679c2c268ba05045332f30db786223f478a5097afbe88544817291 WatchSource:0}: Error finding container d351bff816679c2c268ba05045332f30db786223f478a5097afbe88544817291: Status 404 returned error can't find the container with id d351bff816679c2c268ba05045332f30db786223f478a5097afbe88544817291
Sep 09 15:51:37.927887 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:51:37.927859    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c0dea1_1500_498d_ade5_e8bd61bbcd0b.slice/crio-5e81758e68d1641fb112fa937d4cabd16be7205d2e38e4c10d89537a67636f89 WatchSource:0}: Error finding container 5e81758e68d1641fb112fa937d4cabd16be7205d2e38e4c10d89537a67636f89: Status 404 returned error can't find the container with id 5e81758e68d1641fb112fa937d4cabd16be7205d2e38e4c10d89537a67636f89
Sep 09 15:51:38.050089 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.050022    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:51:38.270405 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.270370    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret\") pod \"global-pull-secret-syncer-7fdsm\" (UID: \"9c5f3c15-9eee-43d1-b403-f89806e864cb\") " pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:38.270594 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.270414    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:38.270594 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.270455    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gsz4\" (UniqueName: \"kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4\") pod \"network-check-target-xsht8\" (UID: \"7bafb10d-a48d-4429-b1a4-532d3ed5a475\") " pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:38.270594 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:38.270541    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:51:38.270594 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:38.270562    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:51:38.270594 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:38.270564    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:51:38.270594 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:38.270578    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:51:38.270594 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:38.270592    2556 projected.go:196] Error preparing data for projected volume kube-api-access-7gsz4 for pod openshift-network-diagnostics/network-check-target-xsht8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:51:38.270845 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:38.270622    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret podName:9c5f3c15-9eee-43d1-b403-f89806e864cb nodeName:}" failed. No retries permitted until 2026-09-09 15:51:39.27060313 +0000 UTC m=+4.137114351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret") pod "global-pull-secret-syncer-7fdsm" (UID: "9c5f3c15-9eee-43d1-b403-f89806e864cb") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:51:38.270845 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:38.270640    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs podName:72246d50-b16d-4f50-b064-aa2a85cf2970 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:39.270632412 +0000 UTC m=+4.137143610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs") pod "network-metrics-daemon-4tmj4" (UID: "72246d50-b16d-4f50-b064-aa2a85cf2970") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:51:38.270845 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:38.270650    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4 podName:7bafb10d-a48d-4429-b1a4-532d3ed5a475 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:39.270645475 +0000 UTC m=+4.137156673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gsz4" (UniqueName: "kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4") pod "network-check-target-xsht8" (UID: "7bafb10d-a48d-4429-b1a4-532d3ed5a475") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:51:38.597490 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.597443    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 15:46:37 +0000 UTC" deadline="2028-03-30 12:22:36.634141645 +0000 UTC"
Sep 09 15:51:38.597905 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.597499    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13628h30m58.036663097s"
Sep 09 15:51:38.757496 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.757442    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:38.757663 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:38.757585    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:51:38.757663 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.757637    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:38.757782 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:38.757762    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:51:38.768942 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.768910    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f47qf" event={"ID":"d8627dab-9db2-4cb5-9ee9-37c644998bbb","Type":"ContainerStarted","Data":"77a3b842b75d181e2b952f471d138fc77c265ac7f41fb056cea308c1e5befb0a"}
Sep 09 15:51:38.770253 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.770188    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kx87f" event={"ID":"c1c0dea1-1500-498d-ade5-e8bd61bbcd0b","Type":"ContainerStarted","Data":"5e81758e68d1641fb112fa937d4cabd16be7205d2e38e4c10d89537a67636f89"}
Sep 09 15:51:38.771524 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.771497    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lxvn6" event={"ID":"cc5d1009-a15f-4c28-b291-074695366f79","Type":"ContainerStarted","Data":"1699d9d82c1a3359bdd7176fb005b403f92bb979637d1488262a8a2f77e1eec5"}
Sep 09 15:51:38.772864 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.772840    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp9wv" event={"ID":"322726ce-594b-4b74-968b-40ad101cb1fb","Type":"ContainerStarted","Data":"ff23b3bc2428679a6683eeadc1860470d0dd2cc465afdd25ddca1ebadbb0da33"}
Sep 09 15:51:38.774113 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.774090    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wv8m7" event={"ID":"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9","Type":"ContainerStarted","Data":"eebfc9bcd8aa0b8591778dd8d6a4d12c2b0830f24dbdeee3fb261ffb0f63c7c4"}
Sep 09 15:51:38.775968 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.775897    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5jrvg" event={"ID":"0880616b-7f54-4880-816e-f21755439fdf","Type":"ContainerStarted","Data":"d351bff816679c2c268ba05045332f30db786223f478a5097afbe88544817291"}
Sep 09 15:51:38.780616 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.780485    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r" event={"ID":"80563802-ee38-4620-addd-926a31290be2","Type":"ContainerStarted","Data":"cc154d619fa33580bf5e187fcbf7b141e4a412f7428ce8e5eda98d7694ee87c0"}
Sep 09 15:51:38.782583 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.782555    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dzktt" event={"ID":"8d08d290-7a84-4502-8534-3c03383e94f0","Type":"ContainerStarted","Data":"21e4586aca403f6100a4556eda5cb885e9fd21c723590d0a8c078c2d9ff8c298"}
Sep 09 15:51:38.784707 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:38.784684    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj" event={"ID":"0a678faa-f9a8-4199-b722-b371b9dc2b59","Type":"ContainerStarted","Data":"39075e9b08e88b9aab2f2119bea5fcce42af6809500f2cfc347ba92c912f8897"}
Sep 09 15:51:39.276878 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:39.276831    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret\") pod \"global-pull-secret-syncer-7fdsm\" (UID: \"9c5f3c15-9eee-43d1-b403-f89806e864cb\") " pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:39.276878 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:39.276882    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:39.277113 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:39.276924    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gsz4\" (UniqueName: \"kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4\") pod \"network-check-target-xsht8\" (UID: \"7bafb10d-a48d-4429-b1a4-532d3ed5a475\") " pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:39.277113 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:39.277006    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:51:39.277113 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:39.277037    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:51:39.277113 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:39.277054    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:51:39.277113 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:39.277069    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:51:39.277113 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:39.277080    2556 projected.go:196] Error preparing data for projected volume kube-api-access-7gsz4 for pod openshift-network-diagnostics/network-check-target-xsht8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:51:39.277113 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:39.277084    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret podName:9c5f3c15-9eee-43d1-b403-f89806e864cb nodeName:}" failed. No retries permitted until 2026-09-09 15:51:41.277062969 +0000 UTC m=+6.143574182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret") pod "global-pull-secret-syncer-7fdsm" (UID: "9c5f3c15-9eee-43d1-b403-f89806e864cb") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:51:39.277113 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:39.277105    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs podName:72246d50-b16d-4f50-b064-aa2a85cf2970 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:41.277094621 +0000 UTC m=+6.143605828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs") pod "network-metrics-daemon-4tmj4" (UID: "72246d50-b16d-4f50-b064-aa2a85cf2970") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:51:39.277113 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:39.277120    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4 podName:7bafb10d-a48d-4429-b1a4-532d3ed5a475 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:41.277111327 +0000 UTC m=+6.143622531 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gsz4" (UniqueName: "kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4") pod "network-check-target-xsht8" (UID: "7bafb10d-a48d-4429-b1a4-532d3ed5a475") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:51:39.598492 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:39.598425    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 15:46:37 +0000 UTC" deadline="2028-02-02 18:43:42.658001103 +0000 UTC"
Sep 09 15:51:39.598851 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:39.598495    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12266h52m3.059511015s"
Sep 09 15:51:39.757602 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:39.757577    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:39.757752 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:39.757727    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:51:39.794703 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:39.794665    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal" event={"ID":"a2ab3a812dc27588fde74d62d7cf30f3","Type":"ContainerStarted","Data":"faea222d7ff4abf08c0aa05e29ccad28ef8499f7ec47d1334cc6608f2558eeb5"}
Sep 09 15:51:40.758550 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:40.757815    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:40.758550 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:40.757951    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:51:40.758550 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:40.758340    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:40.758550 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:40.758439    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:51:40.805711 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:40.805638    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" event={"ID":"2859ad4c0a6499bd80e58f83465fc454","Type":"ContainerStarted","Data":"f05b0170db8525dcc045b1b6e5d19d1d2be2e27a8e45892459937b3a82200754"}
Sep 09 15:51:40.818193 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:40.817146    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-93.ec2.internal" podStartSLOduration=4.817127978 podStartE2EDuration="4.817127978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:40.816778378 +0000 UTC m=+5.683289598" watchObservedRunningTime="2026-09-09 15:51:40.817127978 +0000 UTC m=+5.683639200"
Sep 09 15:51:41.299050 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:41.297824    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gsz4\" (UniqueName: \"kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4\") pod \"network-check-target-xsht8\" (UID: \"7bafb10d-a48d-4429-b1a4-532d3ed5a475\") " pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:41.299050 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:41.297926    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret\") pod \"global-pull-secret-syncer-7fdsm\" (UID: \"9c5f3c15-9eee-43d1-b403-f89806e864cb\") " pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:41.299050 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:41.297959    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:41.299050 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:41.298082    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:51:41.299050 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:41.298142    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs podName:72246d50-b16d-4f50-b064-aa2a85cf2970 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:45.29812431 +0000 UTC m=+10.164635523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs") pod "network-metrics-daemon-4tmj4" (UID: "72246d50-b16d-4f50-b064-aa2a85cf2970") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:51:41.299050 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:41.298825    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:51:41.299050 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:41.298849    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:51:41.299050 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:41.298862    2556 projected.go:196] Error preparing data for projected volume kube-api-access-7gsz4 for pod openshift-network-diagnostics/network-check-target-xsht8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:51:41.299050 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:41.298909    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4 podName:7bafb10d-a48d-4429-b1a4-532d3ed5a475 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:45.298893199 +0000 UTC m=+10.165404411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gsz4" (UniqueName: "kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4") pod "network-check-target-xsht8" (UID: "7bafb10d-a48d-4429-b1a4-532d3ed5a475") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:51:41.299050 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:41.298972    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:51:41.299050 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:41.299009    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret podName:9c5f3c15-9eee-43d1-b403-f89806e864cb nodeName:}" failed. No retries permitted until 2026-09-09 15:51:45.298998464 +0000 UTC m=+10.165509677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret") pod "global-pull-secret-syncer-7fdsm" (UID: "9c5f3c15-9eee-43d1-b403-f89806e864cb") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:51:41.757796 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:41.757664    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:41.757949 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:41.757820    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:51:42.757595 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:42.757558    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:42.758062 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:42.757570    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:42.758062 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:42.757690    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:51:42.758062 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:42.757780    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:51:43.757347 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:43.757320    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:43.757530 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:43.757437    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:51:44.757270 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:44.757232    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:44.757857 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:44.757288    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:44.757857 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:44.757379    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:51:44.757857 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:44.757530    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:51:45.335640 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:45.334716    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret\") pod \"global-pull-secret-syncer-7fdsm\" (UID: \"9c5f3c15-9eee-43d1-b403-f89806e864cb\") " pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:45.335640 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:45.334768    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:45.335640 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:45.334810    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gsz4\" (UniqueName: \"kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4\") pod \"network-check-target-xsht8\" (UID: \"7bafb10d-a48d-4429-b1a4-532d3ed5a475\") " pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:45.335640 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:45.334971    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:51:45.335640 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:45.334992    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:51:45.335640 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:45.335006    2556 projected.go:196] Error preparing data for projected volume kube-api-access-7gsz4 for pod openshift-network-diagnostics/network-check-target-xsht8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:51:45.335640 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:45.335062    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4 podName:7bafb10d-a48d-4429-b1a4-532d3ed5a475 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:53.335045103 +0000 UTC m=+18.201556319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gsz4" (UniqueName: "kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4") pod "network-check-target-xsht8" (UID: "7bafb10d-a48d-4429-b1a4-532d3ed5a475") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:51:45.335640 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:45.335452    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:51:45.335640 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:45.335519    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret podName:9c5f3c15-9eee-43d1-b403-f89806e864cb nodeName:}" failed. No retries permitted until 2026-09-09 15:51:53.335503281 +0000 UTC m=+18.202014494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret") pod "global-pull-secret-syncer-7fdsm" (UID: "9c5f3c15-9eee-43d1-b403-f89806e864cb") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:51:45.335640 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:45.335577    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:51:45.335640 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:45.335607    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs podName:72246d50-b16d-4f50-b064-aa2a85cf2970 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:53.335595815 +0000 UTC m=+18.202107028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs") pod "network-metrics-daemon-4tmj4" (UID: "72246d50-b16d-4f50-b064-aa2a85cf2970") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:51:45.758920 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:45.758833    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:45.759356 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:45.758965    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:51:46.757792 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:46.757748    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:46.757938 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:46.757892    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:51:46.757938 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:46.757759    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:46.758072 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:46.758049    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:51:46.822195 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:46.821895    2556 generic.go:356] "Generic (PLEG): container finished" podID="2859ad4c0a6499bd80e58f83465fc454" containerID="f05b0170db8525dcc045b1b6e5d19d1d2be2e27a8e45892459937b3a82200754" exitCode=0
Sep 09 15:51:46.822195 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:46.821950    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" event={"ID":"2859ad4c0a6499bd80e58f83465fc454","Type":"ContainerDied","Data":"f05b0170db8525dcc045b1b6e5d19d1d2be2e27a8e45892459937b3a82200754"}
Sep 09 15:51:47.757351 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:47.757314    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:47.757548 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:47.757460    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:51:48.758012 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:48.757977    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:48.758012 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:48.758008    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:48.758478 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:48.758104    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:51:48.758478 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:48.758227    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:51:49.758085 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:49.758042    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:49.758578 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:49.758193    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:51:50.757538 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:50.757499    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:50.757710 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:50.757517    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:50.757710 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:50.757635    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:51:50.757710 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:50.757672    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:51:51.757338 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:51.757301    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:51.757716 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:51.757448    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:51:52.757574 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:52.757541    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:52.758105 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:52.757578    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:52.758105 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:52.757671    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:51:52.758105 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:52.757812    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:51:53.396646 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:53.396591    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret\") pod \"global-pull-secret-syncer-7fdsm\" (UID: \"9c5f3c15-9eee-43d1-b403-f89806e864cb\") " pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:53.396821 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:53.396665    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:53.396821 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:53.396717    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gsz4\" (UniqueName: \"kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4\") pod \"network-check-target-xsht8\" (UID: \"7bafb10d-a48d-4429-b1a4-532d3ed5a475\") " pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:53.396821 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:53.396771    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:51:53.397001 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:53.396832    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:51:53.397001 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:53.396848    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret podName:9c5f3c15-9eee-43d1-b403-f89806e864cb nodeName:}" failed. No retries permitted until 2026-09-09 15:52:09.396826679 +0000 UTC m=+34.263337882 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret") pod "global-pull-secret-syncer-7fdsm" (UID: "9c5f3c15-9eee-43d1-b403-f89806e864cb") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:51:53.397001 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:53.396892    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs podName:72246d50-b16d-4f50-b064-aa2a85cf2970 nodeName:}" failed. No retries permitted until 2026-09-09 15:52:09.396873758 +0000 UTC m=+34.263384981 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs") pod "network-metrics-daemon-4tmj4" (UID: "72246d50-b16d-4f50-b064-aa2a85cf2970") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:51:53.397001 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:53.396839    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:51:53.397001 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:53.396927    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:51:53.397001 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:53.396939    2556 projected.go:196] Error preparing data for projected volume kube-api-access-7gsz4 for pod openshift-network-diagnostics/network-check-target-xsht8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:51:53.397001 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:53.396979    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4 podName:7bafb10d-a48d-4429-b1a4-532d3ed5a475 nodeName:}" failed. No retries permitted until 2026-09-09 15:52:09.396968781 +0000 UTC m=+34.263479995 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gsz4" (UniqueName: "kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4") pod "network-check-target-xsht8" (UID: "7bafb10d-a48d-4429-b1a4-532d3ed5a475") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:51:53.758161 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:53.758083    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:53.758570 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:53.758256    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:51:54.757533 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:54.757499    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:54.757733 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:54.757542    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:54.757733 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:54.757626    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:51:54.757853 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:54.757747    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:51:55.758093 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:55.758060    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:55.758560 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:55.758165    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:51:56.758267 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.757868    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:56.759035 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:56.758364    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:51:56.759035 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.757917    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:56.759035 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:56.758835    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:51:56.843733 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.843633    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wv8m7" event={"ID":"63f36a9e-8e7a-49cf-b8f6-635aa633bfa9","Type":"ContainerStarted","Data":"f31af87707acf8aaf5fa52042dd37a69a9514c86c757e665594248adbbc47c07"}
Sep 09 15:51:56.845626 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.845591    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5jrvg" event={"ID":"0880616b-7f54-4880-816e-f21755439fdf","Type":"ContainerStarted","Data":"c5551f6d5792e6306e6ea266cea0bcce884852fad757249a9ef80a3c67b30580"}
Sep 09 15:51:56.849143 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.849122    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 15:51:56.849518 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.849497    2556 generic.go:356] "Generic (PLEG): container finished" podID="80563802-ee38-4620-addd-926a31290be2" containerID="91b6495cb72168422fd861b6d4d7d30882ffaf7d70e941a17976f7c547cff5e0" exitCode=1
Sep 09 15:51:56.849605 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.849571    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r" event={"ID":"80563802-ee38-4620-addd-926a31290be2","Type":"ContainerStarted","Data":"917d15a55d82d88c86cf52bcf61f02375c85fb61b6ab526ff231868a15c393d5"}
Sep 09 15:51:56.849659 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.849647    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r" event={"ID":"80563802-ee38-4620-addd-926a31290be2","Type":"ContainerStarted","Data":"57b6414d027079b98d31527b35ddc6a1b43e4615bdf265ee215c22372e1439b5"}
Sep 09 15:51:56.849711 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.849690    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r" event={"ID":"80563802-ee38-4620-addd-926a31290be2","Type":"ContainerStarted","Data":"b2047305816ec7c8c84951408c63a56c50b829f18430298c6f8ef1def0c1624c"}
Sep 09 15:51:56.849757 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.849720    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r" event={"ID":"80563802-ee38-4620-addd-926a31290be2","Type":"ContainerStarted","Data":"32cccf38d99d33a44ca54e8274f81149edc1d1a7ef9057da14b6e00a04bb3792"}
Sep 09 15:51:56.849806 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.849756    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r" event={"ID":"80563802-ee38-4620-addd-926a31290be2","Type":"ContainerDied","Data":"91b6495cb72168422fd861b6d4d7d30882ffaf7d70e941a17976f7c547cff5e0"}
Sep 09 15:51:56.849806 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.849793    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r" event={"ID":"80563802-ee38-4620-addd-926a31290be2","Type":"ContainerStarted","Data":"9c79d9ee8f793176bd87407ac89fcbab7bf1a9d9a7e27601eb7f71e6b1a9b9df"}
Sep 09 15:51:56.850970 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.850947    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dzktt" event={"ID":"8d08d290-7a84-4502-8534-3c03383e94f0","Type":"ContainerStarted","Data":"73303cc142881564afeb8cd64d618392dcced7b482c98e43ee2e927644f1d3d3"}
Sep 09 15:51:56.852507 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.852488    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj" event={"ID":"0a678faa-f9a8-4199-b722-b371b9dc2b59","Type":"ContainerStarted","Data":"f389048ad3a2ba81ac919b35474982f047cca39056cc9246997f297866bd86ed"}
Sep 09 15:51:56.854311 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.854291    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" event={"ID":"2859ad4c0a6499bd80e58f83465fc454","Type":"ContainerStarted","Data":"f1bac200513f3255075d291bea7f1004012814d36f3e578127c6f4aa804fbd1c"}
Sep 09 15:51:56.856035 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.856005    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kx87f" event={"ID":"c1c0dea1-1500-498d-ade5-e8bd61bbcd0b","Type":"ContainerStarted","Data":"9d3d8d0156e2b69d02bd3723a8d7a27df4abf4d44d93c05492a5779bdc38d00a"}
Sep 09 15:51:56.857851 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.857808    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lxvn6" event={"ID":"cc5d1009-a15f-4c28-b291-074695366f79","Type":"ContainerStarted","Data":"94db2e2b84b815ea98bf5e2b7843a30ae12ac6e226cf2f047e353d1016929272"}
Sep 09 15:51:56.859263 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.859219    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wv8m7" podStartSLOduration=3.790450481 podStartE2EDuration="21.859207531s" totalImagesPullingTime="18.06875705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:56.858697816 +0000 UTC m=+21.725209038" watchObservedRunningTime="2026-09-09 15:51:56.859207531 +0000 UTC m=+21.725718789"
Sep 09 15:51:56.859668 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.859644    2556 generic.go:356] "Generic (PLEG): container finished" podID="322726ce-594b-4b74-968b-40ad101cb1fb" containerID="b16aacead18097a30db772e1299282808dcb820be60b0d9d4cc46ee4764818e8" exitCode=0
Sep 09 15:51:56.859740 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.859681    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp9wv" event={"ID":"322726ce-594b-4b74-968b-40ad101cb1fb","Type":"ContainerDied","Data":"b16aacead18097a30db772e1299282808dcb820be60b0d9d4cc46ee4764818e8"}
Sep 09 15:51:56.871290 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.871251    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-93.ec2.internal" podStartSLOduration=20.871238787 podStartE2EDuration="20.871238787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:56.871016381 +0000 UTC m=+21.737527601" watchObservedRunningTime="2026-09-09 15:51:56.871238787 +0000 UTC m=+21.737750006"
Sep 09 15:51:56.883930 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.883889    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dzktt" podStartSLOduration=3.783296026 podStartE2EDuration="21.883875737s" totalImagesPullingTime="18.100579711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:56.883851277 +0000 UTC m=+21.750362497" watchObservedRunningTime="2026-09-09 15:51:56.883875737 +0000 UTC m=+21.750386957"
Sep 09 15:51:56.898723 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.898670    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lxvn6" podStartSLOduration=3.188600617 podStartE2EDuration="21.898649523s" totalImagesPullingTime="18.710048906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:56.898509214 +0000 UTC m=+21.765020435" watchObservedRunningTime="2026-09-09 15:51:56.898649523 +0000 UTC m=+21.765160756"
Sep 09 15:51:56.912076 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.912033    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5jrvg" podStartSLOduration=11.23475805 podStartE2EDuration="21.91202047s" totalImagesPullingTime="10.67726242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:56.911781851 +0000 UTC m=+21.778293073" watchObservedRunningTime="2026-09-09 15:51:56.91202047 +0000 UTC m=+21.778531690"
Sep 09 15:51:56.924795 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:56.924752    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kx87f" podStartSLOduration=3.863290165 podStartE2EDuration="21.924731836s" totalImagesPullingTime="18.061441671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:56.924313692 +0000 UTC m=+21.790824913" watchObservedRunningTime="2026-09-09 15:51:56.924731836 +0000 UTC m=+21.791243068"
Sep 09 15:51:57.546948 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:57.546920    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 15:51:57.638823 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:57.638703    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T15:51:57.546941966Z","UUID":"696cc802-43b1-408c-9e1d-db238efa767f","Handler":null,"Name":"","Endpoint":""}
Sep 09 15:51:57.640318 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:57.640293    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 15:51:57.640446 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:57.640348    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 15:51:57.757908 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:57.757836    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:57.758081 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:57.757970    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:51:57.864832 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:57.864796    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj" event={"ID":"0a678faa-f9a8-4199-b722-b371b9dc2b59","Type":"ContainerStarted","Data":"8cc5fd83749023e64d44d7e91610d6524c6190bf559c413ba1c09f9198c4286f"}
Sep 09 15:51:57.866515 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:57.866487    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f47qf" event={"ID":"d8627dab-9db2-4cb5-9ee9-37c644998bbb","Type":"ContainerStarted","Data":"6c6a434f53bf84d337b500634902b5eea5bdb991c37fd1ac62c7806109eb1620"}
Sep 09 15:51:58.012832 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:58.012747    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:58.101643 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:58.101605    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:58.355430 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:58.355343    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:58.481381 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:58.481346    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:58.757431 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:58.757407    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:51:58.757550 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:58.757432    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:51:58.757550 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:58.757533    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:51:58.757661 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:58.757597    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:51:58.873344 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:58.873314    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 15:51:58.873882 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:58.873854    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r" event={"ID":"80563802-ee38-4620-addd-926a31290be2","Type":"ContainerStarted","Data":"53696ffd413c9a0370666e771618b91ef0315ac8a561acadcd836f835cb0c1cb"}
Sep 09 15:51:59.757967 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:51:59.757935    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:51:59.758167 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:51:59.758097    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:52:00.188252 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:00.188158    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:00.238169 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:00.238134    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:00.757937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:00.757902    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:52:00.757937 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:00.757922    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:52:00.758156 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:00.758020    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:52:00.758219 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:00.758166    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:52:01.049211 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:01.049118    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:01.580721 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:01.580451    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5jrvg"
Sep 09 15:52:01.581151 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:01.581025    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5jrvg"
Sep 09 15:52:01.595348 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:01.595287    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f47qf" podStartSLOduration=8.482644552 podStartE2EDuration="26.595270425s" totalImagesPullingTime="18.112625873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:57.883505977 +0000 UTC m=+22.750017198" watchObservedRunningTime="2026-09-09 15:52:01.595270425 +0000 UTC m=+26.461781656"
Sep 09 15:52:01.760156 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:01.760137    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:52:01.760258 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:01.760234    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:52:01.883340 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:01.883259    2556 generic.go:356] "Generic (PLEG): container finished" podID="322726ce-594b-4b74-968b-40ad101cb1fb" containerID="c3e807468691a63c8fce9392114b01962aa65a0bd50de1555feeef9958361aa1" exitCode=0
Sep 09 15:52:01.883340 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:01.883322    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp9wv" event={"ID":"322726ce-594b-4b74-968b-40ad101cb1fb","Type":"ContainerDied","Data":"c3e807468691a63c8fce9392114b01962aa65a0bd50de1555feeef9958361aa1"}
Sep 09 15:52:01.887269 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:01.887252    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 15:52:01.887633 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:01.887616    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r" event={"ID":"80563802-ee38-4620-addd-926a31290be2","Type":"ContainerStarted","Data":"e835525c26b5181f439d19747038721f453ddae1b0989af6cc3923a8f7beaaf5"}
Sep 09 15:52:01.887846 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:01.887825    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:52:01.887975 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:01.887897    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:52:01.888039 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:01.888026    2556 scope.go:117] "RemoveContainer" containerID="91b6495cb72168422fd861b6d4d7d30882ffaf7d70e941a17976f7c547cff5e0"
Sep 09 15:52:01.889982 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:01.889958    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj" event={"ID":"0a678faa-f9a8-4199-b722-b371b9dc2b59","Type":"ContainerStarted","Data":"a9262508989384c26720f2af2e0f8398e143a89bb3f7037cfa753313febcad0a"}
Sep 09 15:52:01.903986 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:01.903966    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:52:01.918780 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:01.918743    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6sffj" podStartSLOduration=3.497234438 podStartE2EDuration="26.918728626s" totalImagesPullingTime="23.421494188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:52:01.918526754 +0000 UTC m=+26.785037977" watchObservedRunningTime="2026-09-09 15:52:01.918728626 +0000 UTC m=+26.785239845"
Sep 09 15:52:02.283641 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:02.283546    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:02.374750 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:02.374710    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:02.758032 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:02.757951    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:52:02.758629 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:02.758064    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:52:02.758629 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:02.758116    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:52:02.758629 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:02.758202    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:52:02.895073 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:02.895045    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp9wv" event={"ID":"322726ce-594b-4b74-968b-40ad101cb1fb","Type":"ContainerStarted","Data":"cddc8666b9c2ba277f649880ac5d9d1a02a8aef676300ce8e224bb692b3561fa"}
Sep 09 15:52:02.899842 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:02.899822    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 15:52:02.900269 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:02.900226    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r" event={"ID":"80563802-ee38-4620-addd-926a31290be2","Type":"ContainerStarted","Data":"e9c82741980ff00dbab515e2858333dc761d79b59efbd9fe7f8c481119fb85c9"}
Sep 09 15:52:02.900612 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:02.900590    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:52:02.921064 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:02.921036    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:52:02.936559 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:02.936485    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r" podStartSLOduration=9.825724992 podStartE2EDuration="27.936449045s" totalImagesPullingTime="18.110724053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:52:02.935493789 +0000 UTC m=+27.802005011" watchObservedRunningTime="2026-09-09 15:52:02.936449045 +0000 UTC m=+27.802960267"
Sep 09 15:52:03.220312 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:03.220232    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4tmj4"]
Sep 09 15:52:03.220451 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:03.220384    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:52:03.220556 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:03.220529    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:52:03.223484 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:03.223435    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xsht8"]
Sep 09 15:52:03.223595 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:03.223555    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:52:03.223754 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:03.223649    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:52:03.224099 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:03.224076    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7fdsm"]
Sep 09 15:52:03.224179 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:03.224166    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:52:03.224301 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:03.224278    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:52:03.904868 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:03.904836    2556 generic.go:356] "Generic (PLEG): container finished" podID="322726ce-594b-4b74-968b-40ad101cb1fb" containerID="cddc8666b9c2ba277f649880ac5d9d1a02a8aef676300ce8e224bb692b3561fa" exitCode=0
Sep 09 15:52:03.905571 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:03.904921    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp9wv" event={"ID":"322726ce-594b-4b74-968b-40ad101cb1fb","Type":"ContainerDied","Data":"cddc8666b9c2ba277f649880ac5d9d1a02a8aef676300ce8e224bb692b3561fa"}
Sep 09 15:52:04.427942 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:04.427763    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5jrvg"
Sep 09 15:52:04.428095 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:04.428039    2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 15:52:04.428358 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:04.428339    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5jrvg"
Sep 09 15:52:04.757778 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:04.757682    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:52:04.757778 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:04.757682    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:52:04.757969 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:04.757814    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:52:04.757969 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:04.757682    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:52:04.757969 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:04.757864    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:52:04.757969 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:04.757900    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:52:05.912383 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:05.912288    2556 generic.go:356] "Generic (PLEG): container finished" podID="322726ce-594b-4b74-968b-40ad101cb1fb" containerID="bcdc973d331a84d4af382dd0fbc1c68b4d6b5975161882a879fba746acf2f30f" exitCode=0
Sep 09 15:52:05.912383 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:05.912318    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp9wv" event={"ID":"322726ce-594b-4b74-968b-40ad101cb1fb","Type":"ContainerDied","Data":"bcdc973d331a84d4af382dd0fbc1c68b4d6b5975161882a879fba746acf2f30f"}
Sep 09 15:52:06.757502 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:06.757433    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:52:06.757502 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:06.757433    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:52:06.757750 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:06.757446    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:52:06.757750 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:06.757591    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:52:06.757750 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:06.757661    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsht8" podUID="7bafb10d-a48d-4429-b1a4-532d3ed5a475"
Sep 09 15:52:06.757885 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:06.757746    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7fdsm" podUID="9c5f3c15-9eee-43d1-b403-f89806e864cb"
Sep 09 15:52:07.659085 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:07.659048    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:08.361515 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.361405    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:08.374170 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.374143    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-93.ec2.internal" event="NodeReady"
Sep 09 15:52:08.374307 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.374268    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 15:52:08.428160 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.428124    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9h8tp"]
Sep 09 15:52:08.443753 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.443726    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hdkxg"]
Sep 09 15:52:08.443912 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.443871    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9h8tp"
Sep 09 15:52:08.446023 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.445983    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:08.447163 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.447144    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p7tqp\""
Sep 09 15:52:08.447617 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.447595    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 15:52:08.447617 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.447615    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 15:52:08.449061 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.448602    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 15:52:08.456219 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.456197    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9h8tp"]
Sep 09 15:52:08.456348 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.456233    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hdkxg"]
Sep 09 15:52:08.456348 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.456334    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:08.462774 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.462752    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 15:52:08.462891 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.462798    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5xfw9\""
Sep 09 15:52:08.462998 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.462979    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 15:52:08.604484 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.604431    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6833917f-b72a-4586-98ce-c68895a682f1-config-volume\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:08.604706 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.604492    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8x54\" (UniqueName: \"kubernetes.io/projected/48f47946-24d8-4c4a-9915-8b21f4fe4082-kube-api-access-m8x54\") pod \"ingress-canary-9h8tp\" (UID: \"48f47946-24d8-4c4a-9915-8b21f4fe4082\") " pod="openshift-ingress-canary/ingress-canary-9h8tp"
Sep 09 15:52:08.604706 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.604547    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6lp6\" (UniqueName: \"kubernetes.io/projected/6833917f-b72a-4586-98ce-c68895a682f1-kube-api-access-w6lp6\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:08.604706 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.604571    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:08.604706 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.604612    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48f47946-24d8-4c4a-9915-8b21f4fe4082-cert\") pod \"ingress-canary-9h8tp\" (UID: \"48f47946-24d8-4c4a-9915-8b21f4fe4082\") " pod="openshift-ingress-canary/ingress-canary-9h8tp"
Sep 09 15:52:08.604706 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.604657    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6833917f-b72a-4586-98ce-c68895a682f1-tmp-dir\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:08.706017 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.705937    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48f47946-24d8-4c4a-9915-8b21f4fe4082-cert\") pod \"ingress-canary-9h8tp\" (UID: \"48f47946-24d8-4c4a-9915-8b21f4fe4082\") " pod="openshift-ingress-canary/ingress-canary-9h8tp"
Sep 09 15:52:08.706017 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.706005    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6833917f-b72a-4586-98ce-c68895a682f1-tmp-dir\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:08.706577 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.706168    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6833917f-b72a-4586-98ce-c68895a682f1-config-volume\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:08.706577 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.706206    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8x54\" (UniqueName: \"kubernetes.io/projected/48f47946-24d8-4c4a-9915-8b21f4fe4082-kube-api-access-m8x54\") pod \"ingress-canary-9h8tp\" (UID: \"48f47946-24d8-4c4a-9915-8b21f4fe4082\") " pod="openshift-ingress-canary/ingress-canary-9h8tp"
Sep 09 15:52:08.706577 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.706265    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6lp6\" (UniqueName: \"kubernetes.io/projected/6833917f-b72a-4586-98ce-c68895a682f1-kube-api-access-w6lp6\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:08.706577 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.706294    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:08.706577 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.706328    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6833917f-b72a-4586-98ce-c68895a682f1-tmp-dir\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:08.706577 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:08.706389    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:52:08.706577 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:08.706451    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls podName:6833917f-b72a-4586-98ce-c68895a682f1 nodeName:}" failed. No retries permitted until 2026-09-09 15:52:09.206433536 +0000 UTC m=+34.072944737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls") pod "dns-default-hdkxg" (UID: "6833917f-b72a-4586-98ce-c68895a682f1") : secret "dns-default-metrics-tls" not found
Sep 09 15:52:08.706933 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.706717    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6833917f-b72a-4586-98ce-c68895a682f1-config-volume\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:08.710956 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.710932    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48f47946-24d8-4c4a-9915-8b21f4fe4082-cert\") pod \"ingress-canary-9h8tp\" (UID: \"48f47946-24d8-4c4a-9915-8b21f4fe4082\") " pod="openshift-ingress-canary/ingress-canary-9h8tp"
Sep 09 15:52:08.715203 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.715177    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6lp6\" (UniqueName: \"kubernetes.io/projected/6833917f-b72a-4586-98ce-c68895a682f1-kube-api-access-w6lp6\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:08.715724 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.715698    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8x54\" (UniqueName: \"kubernetes.io/projected/48f47946-24d8-4c4a-9915-8b21f4fe4082-kube-api-access-m8x54\") pod \"ingress-canary-9h8tp\" (UID: \"48f47946-24d8-4c4a-9915-8b21f4fe4082\") " pod="openshift-ingress-canary/ingress-canary-9h8tp"
Sep 09 15:52:08.755556 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.755523    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9h8tp"
Sep 09 15:52:08.757732 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.757706    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:52:08.757860 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.757705    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:52:08.757916 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.757717    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:52:08.760427 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.760400    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 15:52:08.760427 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.760417    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 15:52:08.760744 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.760400    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 15:52:08.761395 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.761379    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 15:52:08.762063 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.761905    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f2dhs\""
Sep 09 15:52:08.762063 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:08.762050    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tvtml\""
Sep 09 15:52:09.211056 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:09.211019    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:09.211321 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:09.211213    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:52:09.211321 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:09.211310    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls podName:6833917f-b72a-4586-98ce-c68895a682f1 nodeName:}" failed. No retries permitted until 2026-09-09 15:52:10.211286945 +0000 UTC m=+35.077798146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls") pod "dns-default-hdkxg" (UID: "6833917f-b72a-4586-98ce-c68895a682f1") : secret "dns-default-metrics-tls" not found
Sep 09 15:52:09.412291 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:09.412255    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gsz4\" (UniqueName: \"kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4\") pod \"network-check-target-xsht8\" (UID: \"7bafb10d-a48d-4429-b1a4-532d3ed5a475\") " pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:52:09.412482 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:09.412427    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret\") pod \"global-pull-secret-syncer-7fdsm\" (UID: \"9c5f3c15-9eee-43d1-b403-f89806e864cb\") " pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:52:09.412561 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:09.412481    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:52:09.412619 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:09.412573    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 15:52:09.412679 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:09.412642    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs podName:72246d50-b16d-4f50-b064-aa2a85cf2970 nodeName:}" failed. No retries permitted until 2026-09-09 15:52:41.41262262 +0000 UTC m=+66.279133826 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs") pod "network-metrics-daemon-4tmj4" (UID: "72246d50-b16d-4f50-b064-aa2a85cf2970") : secret "metrics-daemon-secret" not found
Sep 09 15:52:09.415180 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:09.415157    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c5f3c15-9eee-43d1-b403-f89806e864cb-original-pull-secret\") pod \"global-pull-secret-syncer-7fdsm\" (UID: \"9c5f3c15-9eee-43d1-b403-f89806e864cb\") " pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:52:09.415420 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:09.415397    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gsz4\" (UniqueName: \"kubernetes.io/projected/7bafb10d-a48d-4429-b1a4-532d3ed5a475-kube-api-access-7gsz4\") pod \"network-check-target-xsht8\" (UID: \"7bafb10d-a48d-4429-b1a4-532d3ed5a475\") " pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:52:09.671034 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:09.670934    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:52:09.685128 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:09.685094    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7fdsm"
Sep 09 15:52:09.685346 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:09.685129    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:09.784564 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:09.784528    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:10.221290 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:10.221254    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:10.221413 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:10.221395    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:52:10.221605 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:10.221485    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls podName:6833917f-b72a-4586-98ce-c68895a682f1 nodeName:}" failed. No retries permitted until 2026-09-09 15:52:12.221449182 +0000 UTC m=+37.087960400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls") pod "dns-default-hdkxg" (UID: "6833917f-b72a-4586-98ce-c68895a682f1") : secret "dns-default-metrics-tls" not found
Sep 09 15:52:10.272953 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:10.272922    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xsht8"]
Sep 09 15:52:10.273986 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:10.273864    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7fdsm"]
Sep 09 15:52:10.274830 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:10.274683    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9h8tp"]
Sep 09 15:52:10.276649 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:52:10.276623    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bafb10d_a48d_4429_b1a4_532d3ed5a475.slice/crio-d30b2ac2898ba0466b3ed144e9881b408a776933cf2d5013635aa45d94d78908 WatchSource:0}: Error finding container d30b2ac2898ba0466b3ed144e9881b408a776933cf2d5013635aa45d94d78908: Status 404 returned error can't find the container with id d30b2ac2898ba0466b3ed144e9881b408a776933cf2d5013635aa45d94d78908
Sep 09 15:52:10.277224 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:52:10.277202    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f47946_24d8_4c4a_9915_8b21f4fe4082.slice/crio-5101af7c4d496596fd627bb922b95bbacfb87712862824bdaa18e9d681d1b125 WatchSource:0}: Error finding container 5101af7c4d496596fd627bb922b95bbacfb87712862824bdaa18e9d681d1b125: Status 404 returned error can't find the container with id 5101af7c4d496596fd627bb922b95bbacfb87712862824bdaa18e9d681d1b125
Sep 09 15:52:10.278161 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:52:10.278142    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5f3c15_9eee_43d1_b403_f89806e864cb.slice/crio-d29ea95f9205e08435e012f6ee25ac13288ae5493ce5f076455c697e80425a21 WatchSource:0}: Error finding container d29ea95f9205e08435e012f6ee25ac13288ae5493ce5f076455c697e80425a21: Status 404 returned error can't find the container with id d29ea95f9205e08435e012f6ee25ac13288ae5493ce5f076455c697e80425a21
Sep 09 15:52:10.926506 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:10.926457    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9h8tp" event={"ID":"48f47946-24d8-4c4a-9915-8b21f4fe4082","Type":"ContainerStarted","Data":"5101af7c4d496596fd627bb922b95bbacfb87712862824bdaa18e9d681d1b125"}
Sep 09 15:52:10.929893 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:10.929860    2556 generic.go:356] "Generic (PLEG): container finished" podID="322726ce-594b-4b74-968b-40ad101cb1fb" containerID="7ba92a05218cfe2adfa22bd103f1335f6f337b31cd7e8bd0e527b411f69d3abb" exitCode=0
Sep 09 15:52:10.930025 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:10.929937    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp9wv" event={"ID":"322726ce-594b-4b74-968b-40ad101cb1fb","Type":"ContainerDied","Data":"7ba92a05218cfe2adfa22bd103f1335f6f337b31cd7e8bd0e527b411f69d3abb"}
Sep 09 15:52:10.931375 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:10.931344    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xsht8" event={"ID":"7bafb10d-a48d-4429-b1a4-532d3ed5a475","Type":"ContainerStarted","Data":"d30b2ac2898ba0466b3ed144e9881b408a776933cf2d5013635aa45d94d78908"}
Sep 09 15:52:10.932682 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:10.932657    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7fdsm" event={"ID":"9c5f3c15-9eee-43d1-b403-f89806e864cb","Type":"ContainerStarted","Data":"d29ea95f9205e08435e012f6ee25ac13288ae5493ce5f076455c697e80425a21"}
Sep 09 15:52:11.940037 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:11.939997    2556 generic.go:356] "Generic (PLEG): container finished" podID="322726ce-594b-4b74-968b-40ad101cb1fb" containerID="d5560b89219ab63bb124f5e82b246b18bb6fb9a05ee206aa930e7122226a4290" exitCode=0
Sep 09 15:52:11.940573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:11.940048    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp9wv" event={"ID":"322726ce-594b-4b74-968b-40ad101cb1fb","Type":"ContainerDied","Data":"d5560b89219ab63bb124f5e82b246b18bb6fb9a05ee206aa930e7122226a4290"}
Sep 09 15:52:12.241343 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:12.241005    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:12.241343 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:12.241171    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:52:12.241343 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:12.241344    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls podName:6833917f-b72a-4586-98ce-c68895a682f1 nodeName:}" failed. No retries permitted until 2026-09-09 15:52:16.241322103 +0000 UTC m=+41.107833319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls") pod "dns-default-hdkxg" (UID: "6833917f-b72a-4586-98ce-c68895a682f1") : secret "dns-default-metrics-tls" not found
Sep 09 15:52:12.949802 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:12.949704    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp9wv" event={"ID":"322726ce-594b-4b74-968b-40ad101cb1fb","Type":"ContainerStarted","Data":"dbc0aec807673fa5dd8516cb16e1efa17b3c49569bb44d12ad6ec9e18843a8a4"}
Sep 09 15:52:12.970241 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:12.970157    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xp9wv" podStartSLOduration=8.591832292 podStartE2EDuration="37.970139914s" totalImagesPullingTime="29.378307622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:52:12.968284988 +0000 UTC m=+37.834796228" watchObservedRunningTime="2026-09-09 15:52:12.970139914 +0000 UTC m=+37.836651133"
Sep 09 15:52:13.096648 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:13.096562    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:13.150743 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:13.150706    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:14.382386 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:14.382346    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:14.429448 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:14.429418    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:16.272977 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:16.272937    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:16.273442 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:16.273072    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:52:16.273442 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:16.273133    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls podName:6833917f-b72a-4586-98ce-c68895a682f1 nodeName:}" failed. No retries permitted until 2026-09-09 15:52:24.273118774 +0000 UTC m=+49.139629986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls") pod "dns-default-hdkxg" (UID: "6833917f-b72a-4586-98ce-c68895a682f1") : secret "dns-default-metrics-tls" not found
Sep 09 15:52:16.961158 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:16.960797    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xsht8" event={"ID":"7bafb10d-a48d-4429-b1a4-532d3ed5a475","Type":"ContainerStarted","Data":"945e00f7c866e65044883360dae22d36488eca0374b0767470ca83cdcd8535f6"}
Sep 09 15:52:16.961812 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:16.961786    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:52:16.963931 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:16.963903    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9h8tp" event={"ID":"48f47946-24d8-4c4a-9915-8b21f4fe4082","Type":"ContainerStarted","Data":"81860e4cb6e3d68a71e3dd0884de1c16d010879e89778d19946b9f2387611cde"}
Sep 09 15:52:16.974701 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:16.974655    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xsht8" podStartSLOduration=35.459907961 podStartE2EDuration="41.974636091s" totalImagesPullingTime="6.51472813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:52:16.974012786 +0000 UTC m=+41.840524017" watchObservedRunningTime="2026-09-09 15:52:16.974636091 +0000 UTC m=+41.841147315"
Sep 09 15:52:16.987712 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:16.987671    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9h8tp" podStartSLOduration=2.473448581 podStartE2EDuration="8.987655742s" totalImagesPullingTime="6.514207161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:52:16.986832932 +0000 UTC m=+41.853344155" watchObservedRunningTime="2026-09-09 15:52:16.987655742 +0000 UTC m=+41.854166961"
Sep 09 15:52:17.968227 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:17.968190    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7fdsm" event={"ID":"9c5f3c15-9eee-43d1-b403-f89806e864cb","Type":"ContainerStarted","Data":"20344eb878f0dcfa999179a28ccccff01f7d09ca611b918978e4f9f8ab94463f"}
Sep 09 15:52:21.470672 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:21.470629    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:21.530313 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:21.530278    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:24.330682 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:24.330641    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:24.331170 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:24.330833    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:52:24.331170 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:24.330933    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls podName:6833917f-b72a-4586-98ce-c68895a682f1 nodeName:}" failed. No retries permitted until 2026-09-09 15:52:40.33091044 +0000 UTC m=+65.197421654 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls") pod "dns-default-hdkxg" (UID: "6833917f-b72a-4586-98ce-c68895a682f1") : secret "dns-default-metrics-tls" not found
Sep 09 15:52:27.663477 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:27.663425    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:30.529143 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:30.529107    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:34.918270 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:34.918244    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms57r"
Sep 09 15:52:34.942787 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:34.942737    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7fdsm" podStartSLOduration=53.415812143 podStartE2EDuration="59.942717494s" totalImagesPullingTime="6.526905351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:52:17.982246844 +0000 UTC m=+42.848758064" watchObservedRunningTime="2026-09-09 15:52:34.942717494 +0000 UTC m=+59.809228713"
Sep 09 15:52:35.030729 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:35.030698    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:35.134676 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:35.132460    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:38.135328 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.135288    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:38.195242 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.195206    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:38.763157 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.763127    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr"]
Sep 09 15:52:38.765951 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.765933    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr"
Sep 09 15:52:38.768732 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.768711    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rmswr\""
Sep 09 15:52:38.768878 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.768721    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 15:52:38.769549 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.769525    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 15:52:38.769549 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.769540    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 15:52:38.769823 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.769808    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 15:52:38.774893 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.774875    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr"]
Sep 09 15:52:38.787153 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.787132    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"]
Sep 09 15:52:38.789001 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.788981    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.791423 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.791402    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 15:52:38.791522 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.791430    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 15:52:38.791522 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.791437    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 15:52:38.791522 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.791436    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 15:52:38.800861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.800842    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"]
Sep 09 15:52:38.824524 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.824500    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kzfp\" (UniqueName: \"kubernetes.io/projected/47278756-4b7f-42be-861b-5faf5c764a30-kube-api-access-7kzfp\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.824624 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.824554    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/47278756-4b7f-42be-861b-5faf5c764a30-ca\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.824624 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.824573    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/47278756-4b7f-42be-861b-5faf5c764a30-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.824624 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.824610    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/47278756-4b7f-42be-861b-5faf5c764a30-hub\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.824772 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.824647    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/47278756-4b7f-42be-861b-5faf5c764a30-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.824772 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.824684    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfmgm\" (UniqueName: \"kubernetes.io/projected/f679d979-2107-44f0-b2fd-edf22f859957-kube-api-access-hfmgm\") pod \"managed-serviceaccount-addon-agent-5df7966d8b-6vlzr\" (UID: \"f679d979-2107-44f0-b2fd-edf22f859957\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr"
Sep 09 15:52:38.824772 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.824713    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47278756-4b7f-42be-861b-5faf5c764a30-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.824861 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.824768    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f679d979-2107-44f0-b2fd-edf22f859957-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5df7966d8b-6vlzr\" (UID: \"f679d979-2107-44f0-b2fd-edf22f859957\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr"
Sep 09 15:52:38.924994 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.924969    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kzfp\" (UniqueName: \"kubernetes.io/projected/47278756-4b7f-42be-861b-5faf5c764a30-kube-api-access-7kzfp\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.925087 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.925044    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/47278756-4b7f-42be-861b-5faf5c764a30-ca\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.925087 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.925062    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/47278756-4b7f-42be-861b-5faf5c764a30-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.925087 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.925079    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/47278756-4b7f-42be-861b-5faf5c764a30-hub\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.925227 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.925094    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/47278756-4b7f-42be-861b-5faf5c764a30-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.925227 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.925113    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfmgm\" (UniqueName: \"kubernetes.io/projected/f679d979-2107-44f0-b2fd-edf22f859957-kube-api-access-hfmgm\") pod \"managed-serviceaccount-addon-agent-5df7966d8b-6vlzr\" (UID: \"f679d979-2107-44f0-b2fd-edf22f859957\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr"
Sep 09 15:52:38.925227 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.925133    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47278756-4b7f-42be-861b-5faf5c764a30-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.925227 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.925167    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f679d979-2107-44f0-b2fd-edf22f859957-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5df7966d8b-6vlzr\" (UID: \"f679d979-2107-44f0-b2fd-edf22f859957\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr"
Sep 09 15:52:38.925800 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.925770    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/47278756-4b7f-42be-861b-5faf5c764a30-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.928720 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.928688    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/47278756-4b7f-42be-861b-5faf5c764a30-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.928809 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.928795    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47278756-4b7f-42be-861b-5faf5c764a30-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.928852 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.928800    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/47278756-4b7f-42be-861b-5faf5c764a30-ca\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.928852 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.928836    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/47278756-4b7f-42be-861b-5faf5c764a30-hub\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.928929 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.928912    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f679d979-2107-44f0-b2fd-edf22f859957-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5df7966d8b-6vlzr\" (UID: \"f679d979-2107-44f0-b2fd-edf22f859957\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr"
Sep 09 15:52:38.931993 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.931968    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kzfp\" (UniqueName: \"kubernetes.io/projected/47278756-4b7f-42be-861b-5faf5c764a30-kube-api-access-7kzfp\") pod \"cluster-proxy-proxy-agent-7694b94fc9-xmqln\" (UID: \"47278756-4b7f-42be-861b-5faf5c764a30\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:38.932291 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:38.932270    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfmgm\" (UniqueName: \"kubernetes.io/projected/f679d979-2107-44f0-b2fd-edf22f859957-kube-api-access-hfmgm\") pod \"managed-serviceaccount-addon-agent-5df7966d8b-6vlzr\" (UID: \"f679d979-2107-44f0-b2fd-edf22f859957\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr"
Sep 09 15:52:39.087908 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:39.087825    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr"
Sep 09 15:52:39.097083 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:39.097057    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:52:39.352943 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:39.352855    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr"]
Sep 09 15:52:39.357507 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:52:39.357482    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf679d979_2107_44f0_b2fd_edf22f859957.slice/crio-8109ded39fcfafaf29d06a8e52ec3c1e91e0e0b984922837f0ea3c2d1c824edd WatchSource:0}: Error finding container 8109ded39fcfafaf29d06a8e52ec3c1e91e0e0b984922837f0ea3c2d1c824edd: Status 404 returned error can't find the container with id 8109ded39fcfafaf29d06a8e52ec3c1e91e0e0b984922837f0ea3c2d1c824edd
Sep 09 15:52:39.365681 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:39.365655    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"]
Sep 09 15:52:39.369045 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:52:39.369021    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47278756_4b7f_42be_861b_5faf5c764a30.slice/crio-5ceca917150078372367fcbffb294925857b603784184c007a9e8cc60683d1aa WatchSource:0}: Error finding container 5ceca917150078372367fcbffb294925857b603784184c007a9e8cc60683d1aa: Status 404 returned error can't find the container with id 5ceca917150078372367fcbffb294925857b603784184c007a9e8cc60683d1aa
Sep 09 15:52:40.021208 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:40.021176    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln" event={"ID":"47278756-4b7f-42be-861b-5faf5c764a30","Type":"ContainerStarted","Data":"5ceca917150078372367fcbffb294925857b603784184c007a9e8cc60683d1aa"}
Sep 09 15:52:40.022228 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:40.022207    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr" event={"ID":"f679d979-2107-44f0-b2fd-edf22f859957","Type":"ContainerStarted","Data":"8109ded39fcfafaf29d06a8e52ec3c1e91e0e0b984922837f0ea3c2d1c824edd"}
Sep 09 15:52:40.335606 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:40.335521    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:52:40.335739 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:40.335669    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:52:40.335739 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:40.335727    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls podName:6833917f-b72a-4586-98ce-c68895a682f1 nodeName:}" failed. No retries permitted until 2026-09-09 15:53:12.335713536 +0000 UTC m=+97.202224739 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls") pod "dns-default-hdkxg" (UID: "6833917f-b72a-4586-98ce-c68895a682f1") : secret "dns-default-metrics-tls" not found
Sep 09 15:52:41.442182 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:41.442149    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:52:41.442710 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:41.442308    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 15:52:41.442710 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:52:41.442377    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs podName:72246d50-b16d-4f50-b064-aa2a85cf2970 nodeName:}" failed. No retries permitted until 2026-09-09 15:53:45.442361749 +0000 UTC m=+130.308872964 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs") pod "network-metrics-daemon-4tmj4" (UID: "72246d50-b16d-4f50-b064-aa2a85cf2970") : secret "metrics-daemon-secret" not found
Sep 09 15:52:43.031959 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:43.031920    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr" event={"ID":"f679d979-2107-44f0-b2fd-edf22f859957","Type":"ContainerStarted","Data":"d09ea964f11e3e804542b5f486291cecba8d6f058150e8eaf342d171731f27ec"}
Sep 09 15:52:43.048497 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:43.048427    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr" podStartSLOduration=2.170648666 podStartE2EDuration="5.048411031s" totalImagesPullingTime="2.877762365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:52:43.047766672 +0000 UTC m=+67.914277893" watchObservedRunningTime="2026-09-09 15:52:43.048411031 +0000 UTC m=+67.914922252"
Sep 09 15:52:47.047802 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:47.047765    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln" event={"ID":"47278756-4b7f-42be-861b-5faf5c764a30","Type":"ContainerStarted","Data":"fe41aa4de4b6ea05fb82894b5dc4047c01eacf755fbf2016628faa3dd3dd10b5"}
Sep 09 15:52:47.047802 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:47.047806    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln" event={"ID":"47278756-4b7f-42be-861b-5faf5c764a30","Type":"ContainerStarted","Data":"68a6fb203d2f9c41357f8b54d72830cb14dabdcbc89924ecf53aaeab9c4a0ced"}
Sep 09 15:52:47.048355 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:47.047818    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln" event={"ID":"47278756-4b7f-42be-861b-5faf5c764a30","Type":"ContainerStarted","Data":"985e1db3b6d19497d4f257daaee6ffd306d52be1f68fabec04bda79ce6332df2"}
Sep 09 15:52:47.066279 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:47.066227    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln" podStartSLOduration=2.249901488 podStartE2EDuration="9.066210763s" totalImagesPullingTime="6.816309275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:52:47.065406513 +0000 UTC m=+71.931917732" watchObservedRunningTime="2026-09-09 15:52:47.066210763 +0000 UTC m=+71.932721980"
Sep 09 15:52:47.971199 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:52:47.971167    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xsht8"
Sep 09 15:53:07.371793 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:53:07.371670    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:53:12.348135 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:53:12.348098    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:53:12.348576 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:53:12.348216    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:53:12.348576 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:53:12.348272    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls podName:6833917f-b72a-4586-98ce-c68895a682f1 nodeName:}" failed. No retries permitted until 2026-09-09 15:54:16.348258198 +0000 UTC m=+161.214769396 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls") pod "dns-default-hdkxg" (UID: "6833917f-b72a-4586-98ce-c68895a682f1") : secret "dns-default-metrics-tls" not found
Sep 09 15:53:45.480327 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:53:45.480268    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:53:45.480868 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:53:45.480419    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 15:53:45.480868 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:53:45.480524    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs podName:72246d50-b16d-4f50-b064-aa2a85cf2970 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:47.480491218 +0000 UTC m=+252.347002444 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs") pod "network-metrics-daemon-4tmj4" (UID: "72246d50-b16d-4f50-b064-aa2a85cf2970") : secret "metrics-daemon-secret" not found
Sep 09 15:54:11.464933 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:54:11.464876    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hdkxg" podUID="6833917f-b72a-4586-98ce-c68895a682f1"
Sep 09 15:54:11.779032 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:54:11.778944    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:54:12.294270 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:54:12.294242    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:54:16.391738 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:54:16.391688    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:54:16.392156 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:54:16.391845    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:54:16.392156 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:54:16.391908    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls podName:6833917f-b72a-4586-98ce-c68895a682f1 nodeName:}" failed. No retries permitted until 2026-09-09 15:56:18.391892605 +0000 UTC m=+283.258403802 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls") pod "dns-default-hdkxg" (UID: "6833917f-b72a-4586-98ce-c68895a682f1") : secret "dns-default-metrics-tls" not found
Sep 09 15:54:22.757804 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:54:22.757762    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:54:42.380759 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:54:42.380730    2556 generic.go:356] "Generic (PLEG): container finished" podID="f679d979-2107-44f0-b2fd-edf22f859957" containerID="d09ea964f11e3e804542b5f486291cecba8d6f058150e8eaf342d171731f27ec" exitCode=255
Sep 09 15:54:42.381089 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:54:42.380804    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr" event={"ID":"f679d979-2107-44f0-b2fd-edf22f859957","Type":"ContainerDied","Data":"d09ea964f11e3e804542b5f486291cecba8d6f058150e8eaf342d171731f27ec"}
Sep 09 15:54:42.381136 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:54:42.381106    2556 scope.go:117] "RemoveContainer" containerID="d09ea964f11e3e804542b5f486291cecba8d6f058150e8eaf342d171731f27ec"
Sep 09 15:54:43.385280 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:54:43.385244    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df7966d8b-6vlzr" event={"ID":"f679d979-2107-44f0-b2fd-edf22f859957","Type":"ContainerStarted","Data":"18856586bc78b6252ad172d71327f6143df043688b89ed10b9224a53553d59d4"}
Sep 09 15:55:08.218057 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:08.218025    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:38.910695 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:38.910651    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:39.011224 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:39.011192    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:40.970961 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:40.970923    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:41.036578 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:41.036545    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:43.040881 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.040842    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:43.134921 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.134889    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:43.183557 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.183528    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-48skw"]
Sep 09 15:55:43.186444 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.186427    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.189156 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.189132    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 15:55:43.189260 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.189177    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 15:55:43.189260 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.189193    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 15:55:43.189617 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.189596    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rqq2h\""
Sep 09 15:55:43.189617 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.189613    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 15:55:43.197686 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.197666    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-48skw"]
Sep 09 15:55:43.220154 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.220133    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:43.271160 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.271137    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3337cbb2-24ee-40cd-8934-2f11d0069e0f-crio-socket\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.271283 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.271201    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.271283 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.271253    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3337cbb2-24ee-40cd-8934-2f11d0069e0f-data-volume\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.271283 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.271274    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.271379 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.271290    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkbkt\" (UniqueName: \"kubernetes.io/projected/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-api-access-gkbkt\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.371631 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.371563    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3337cbb2-24ee-40cd-8934-2f11d0069e0f-data-volume\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.371631 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.371593    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.371631 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.371610    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbkt\" (UniqueName: \"kubernetes.io/projected/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-api-access-gkbkt\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.371631 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.371631    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3337cbb2-24ee-40cd-8934-2f11d0069e0f-crio-socket\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.371878 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:43.371692    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 15:55:43.371878 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.371720    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.371878 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:43.371763    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm podName:3337cbb2-24ee-40cd-8934-2f11d0069e0f nodeName:}" failed. No retries permitted until 2026-09-09 15:55:43.871745138 +0000 UTC m=+248.738256340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48skw" (UID: "3337cbb2-24ee-40cd-8934-2f11d0069e0f") : configmap "kube-rbac-proxy" not found
Sep 09 15:55:43.371878 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.371808    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3337cbb2-24ee-40cd-8934-2f11d0069e0f-crio-socket\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.371878 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:43.371812    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:55:43.372058 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:43.371888    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls podName:3337cbb2-24ee-40cd-8934-2f11d0069e0f nodeName:}" failed. No retries permitted until 2026-09-09 15:55:43.871869485 +0000 UTC m=+248.738380696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48skw" (UID: "3337cbb2-24ee-40cd-8934-2f11d0069e0f") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:55:43.372058 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.371933    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3337cbb2-24ee-40cd-8934-2f11d0069e0f-data-volume\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.379445 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.379425    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkbkt\" (UniqueName: \"kubernetes.io/projected/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-api-access-gkbkt\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.493401 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.493371    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:43.550671 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.550639    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:43.875674 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.875647    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.875847 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.875710    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:43.875847 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:43.875760    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 15:55:43.875847 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:43.875793    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:55:43.875847 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:43.875823    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm podName:3337cbb2-24ee-40cd-8934-2f11d0069e0f nodeName:}" failed. No retries permitted until 2026-09-09 15:55:44.87580635 +0000 UTC m=+249.742317568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48skw" (UID: "3337cbb2-24ee-40cd-8934-2f11d0069e0f") : configmap "kube-rbac-proxy" not found
Sep 09 15:55:43.875847 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:43.875840    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls podName:3337cbb2-24ee-40cd-8934-2f11d0069e0f nodeName:}" failed. No retries permitted until 2026-09-09 15:55:44.8758289 +0000 UTC m=+249.742340098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48skw" (UID: "3337cbb2-24ee-40cd-8934-2f11d0069e0f") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:55:43.926978 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:43.926947    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:44.245486 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:44.245432    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:44.300272 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:44.300242    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:44.881371 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:44.881330    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:44.881576 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:44.881406    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:44.881576 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:44.881491    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 15:55:44.881576 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:44.881515    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:55:44.881576 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:44.881568    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm podName:3337cbb2-24ee-40cd-8934-2f11d0069e0f nodeName:}" failed. No retries permitted until 2026-09-09 15:55:46.881549952 +0000 UTC m=+251.748061154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48skw" (UID: "3337cbb2-24ee-40cd-8934-2f11d0069e0f") : configmap "kube-rbac-proxy" not found
Sep 09 15:55:44.881717 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:44.881582    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls podName:3337cbb2-24ee-40cd-8934-2f11d0069e0f nodeName:}" failed. No retries permitted until 2026-09-09 15:55:46.881576364 +0000 UTC m=+251.748087562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48skw" (UID: "3337cbb2-24ee-40cd-8934-2f11d0069e0f") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:55:45.742252 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.742225    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dzktt_8d08d290-7a84-4502-8534-3c03383e94f0/dns-node-resolver/0.log"
Sep 09 15:55:45.857512 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.857455    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-lzcjx"]
Sep 09 15:55:45.860296 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.860271    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-lzcjx"
Sep 09 15:55:45.863383 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.863362    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xr8gk\""
Sep 09 15:55:45.864218 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.864195    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 15:55:45.864333 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.864240    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 15:55:45.864333 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.864273    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 15:55:45.864479 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.864348    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 15:55:45.870279 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.870260    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-lzcjx"]
Sep 09 15:55:45.888177 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.888153    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c323ba17-60f9-4a5f-8388-5a91f2092016-signing-key\") pod \"service-ca-75d99c484-lzcjx\" (UID: \"c323ba17-60f9-4a5f-8388-5a91f2092016\") " pod="openshift-service-ca/service-ca-75d99c484-lzcjx"
Sep 09 15:55:45.888305 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.888184    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8s8d\" (UniqueName: \"kubernetes.io/projected/c323ba17-60f9-4a5f-8388-5a91f2092016-kube-api-access-b8s8d\") pod \"service-ca-75d99c484-lzcjx\" (UID: \"c323ba17-60f9-4a5f-8388-5a91f2092016\") " pod="openshift-service-ca/service-ca-75d99c484-lzcjx"
Sep 09 15:55:45.888305 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.888254    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c323ba17-60f9-4a5f-8388-5a91f2092016-signing-cabundle\") pod \"service-ca-75d99c484-lzcjx\" (UID: \"c323ba17-60f9-4a5f-8388-5a91f2092016\") " pod="openshift-service-ca/service-ca-75d99c484-lzcjx"
Sep 09 15:55:45.888565 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.888533    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:45.986840 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.986812    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:45.988765 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.988745    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c323ba17-60f9-4a5f-8388-5a91f2092016-signing-key\") pod \"service-ca-75d99c484-lzcjx\" (UID: \"c323ba17-60f9-4a5f-8388-5a91f2092016\") " pod="openshift-service-ca/service-ca-75d99c484-lzcjx"
Sep 09 15:55:45.988834 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.988772    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8s8d\" (UniqueName: \"kubernetes.io/projected/c323ba17-60f9-4a5f-8388-5a91f2092016-kube-api-access-b8s8d\") pod \"service-ca-75d99c484-lzcjx\" (UID: \"c323ba17-60f9-4a5f-8388-5a91f2092016\") " pod="openshift-service-ca/service-ca-75d99c484-lzcjx"
Sep 09 15:55:45.988980 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.988965    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c323ba17-60f9-4a5f-8388-5a91f2092016-signing-cabundle\") pod \"service-ca-75d99c484-lzcjx\" (UID: \"c323ba17-60f9-4a5f-8388-5a91f2092016\") " pod="openshift-service-ca/service-ca-75d99c484-lzcjx"
Sep 09 15:55:45.989552 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.989532    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c323ba17-60f9-4a5f-8388-5a91f2092016-signing-cabundle\") pod \"service-ca-75d99c484-lzcjx\" (UID: \"c323ba17-60f9-4a5f-8388-5a91f2092016\") " pod="openshift-service-ca/service-ca-75d99c484-lzcjx"
Sep 09 15:55:45.991178 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.991159    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c323ba17-60f9-4a5f-8388-5a91f2092016-signing-key\") pod \"service-ca-75d99c484-lzcjx\" (UID: \"c323ba17-60f9-4a5f-8388-5a91f2092016\") " pod="openshift-service-ca/service-ca-75d99c484-lzcjx"
Sep 09 15:55:45.997123 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:45.997078    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8s8d\" (UniqueName: \"kubernetes.io/projected/c323ba17-60f9-4a5f-8388-5a91f2092016-kube-api-access-b8s8d\") pod \"service-ca-75d99c484-lzcjx\" (UID: \"c323ba17-60f9-4a5f-8388-5a91f2092016\") " pod="openshift-service-ca/service-ca-75d99c484-lzcjx"
Sep 09 15:55:46.168556 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:46.168523    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-lzcjx"
Sep 09 15:55:46.437030 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:46.436993    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-lzcjx"]
Sep 09 15:55:46.443185 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:55:46.443159    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc323ba17_60f9_4a5f_8388_5a91f2092016.slice/crio-052e26a9c5cd2100abf0f2f3ab22be8fcad31e480ba0253e71f86cf076bc75d7 WatchSource:0}: Error finding container 052e26a9c5cd2100abf0f2f3ab22be8fcad31e480ba0253e71f86cf076bc75d7: Status 404 returned error can't find the container with id 052e26a9c5cd2100abf0f2f3ab22be8fcad31e480ba0253e71f86cf076bc75d7
Sep 09 15:55:46.569668 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:46.569634    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-lzcjx" event={"ID":"c323ba17-60f9-4a5f-8388-5a91f2092016","Type":"ContainerStarted","Data":"052e26a9c5cd2100abf0f2f3ab22be8fcad31e480ba0253e71f86cf076bc75d7"}
Sep 09 15:55:46.742949 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:46.742878    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kx87f_c1c0dea1-1500-498d-ade5-e8bd61bbcd0b/node-ca/0.log"
Sep 09 15:55:46.802987 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:46.802953    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:46.897125 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:46.897080    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:46.897286 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:46.897188    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:46.897286 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:46.897217    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 15:55:46.897395 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:46.897298    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm podName:3337cbb2-24ee-40cd-8934-2f11d0069e0f nodeName:}" failed. No retries permitted until 2026-09-09 15:55:50.897276455 +0000 UTC m=+255.763787657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48skw" (UID: "3337cbb2-24ee-40cd-8934-2f11d0069e0f") : configmap "kube-rbac-proxy" not found
Sep 09 15:55:46.897395 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:46.897299    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:55:46.897395 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:46.897347    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls podName:3337cbb2-24ee-40cd-8934-2f11d0069e0f nodeName:}" failed. No retries permitted until 2026-09-09 15:55:50.89732906 +0000 UTC m=+255.763840265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48skw" (UID: "3337cbb2-24ee-40cd-8934-2f11d0069e0f") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:55:47.502460 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:47.502425    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:55:47.502647 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:47.502573    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 15:55:47.502647 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:47.502638    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs podName:72246d50-b16d-4f50-b064-aa2a85cf2970 nodeName:}" failed. No retries permitted until 2026-09-09 15:57:49.502621052 +0000 UTC m=+374.369132250 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs") pod "network-metrics-daemon-4tmj4" (UID: "72246d50-b16d-4f50-b064-aa2a85cf2970") : secret "metrics-daemon-secret" not found
Sep 09 15:55:47.542633 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:47.542599    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9h8tp_48f47946-24d8-4c4a-9915-8b21f4fe4082/serve-healthcheck-canary/0.log"
Sep 09 15:55:48.577239 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:48.577154    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-lzcjx" event={"ID":"c323ba17-60f9-4a5f-8388-5a91f2092016","Type":"ContainerStarted","Data":"e903a3e431e2593a49ed16d7688a50d7bb7aca67d9c59bd140e88baca2a17c52"}
Sep 09 15:55:48.593907 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:48.593507    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-lzcjx" podStartSLOduration=1.830998958 podStartE2EDuration="3.593488142s" totalImagesPullingTime="1.762489184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:55:48.592570165 +0000 UTC m=+253.459081398" watchObservedRunningTime="2026-09-09 15:55:48.593488142 +0000 UTC m=+253.459999364"
Sep 09 15:55:48.909894 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:48.909796    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:49.488437 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:49.488404    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:49.591245 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:49.591210    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:49.691648 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:49.691614    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:49.791900 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:49.791803    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:49.921029 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:49.920997    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:50.077054 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:50.076974    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:50.192786 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:50.192747    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:50.930649 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:50.930615    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:50.931048 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:50.930660    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:50.931048 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:50.930755    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:55:50.931048 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:50.930814    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls podName:3337cbb2-24ee-40cd-8934-2f11d0069e0f nodeName:}" failed. No retries permitted until 2026-09-09 15:55:58.930798359 +0000 UTC m=+263.797309557 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48skw" (UID: "3337cbb2-24ee-40cd-8934-2f11d0069e0f") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:55:50.931048 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:50.930760    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 15:55:50.931048 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:50.930848    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm podName:3337cbb2-24ee-40cd-8934-2f11d0069e0f nodeName:}" failed. No retries permitted until 2026-09-09 15:55:58.930841861 +0000 UTC m=+263.797353059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48skw" (UID: "3337cbb2-24ee-40cd-8934-2f11d0069e0f") : configmap "kube-rbac-proxy" not found
Sep 09 15:55:51.390572 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:51.390487    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:51.584982 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:51.584941    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:51.785115 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:51.785075    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:51.994106 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:51.994069    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:52.192766 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:52.192711    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:52.990975 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:52.990940    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:53.377867 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:53.377777    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:53.431815 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:53.431783    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:53.788341 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:53.788304    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:53.988797 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:53.988765    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:54.278618 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:54.278582    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:54.593203 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:54.593119    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:55.188651 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:55.188612    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:55.391546 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:55.391505    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:55.789973 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:55.789936    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:55.968404 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:55.968370    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:56.069179 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:56.069093    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:56.385352 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:56.385266    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:56.993935 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:56.993904    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:57.394150 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:57.394063    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:57.674818 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:57.674780    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:57.795187 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:57.795116    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:57.984774 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:57.984678    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:58.402743 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:58.402655    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:58.542929 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:58.542880    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:58.995679 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:58.995632    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:58.995679 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:58.995689    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:58.995866 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:58.995776    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:55:58.995915 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:55:58.995885    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls podName:3337cbb2-24ee-40cd-8934-2f11d0069e0f nodeName:}" failed. No retries permitted until 2026-09-09 15:56:14.99586857 +0000 UTC m=+279.862379772 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48skw" (UID: "3337cbb2-24ee-40cd-8934-2f11d0069e0f") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:55:58.996192 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:58.996176    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3337cbb2-24ee-40cd-8934-2f11d0069e0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:55:59.190728 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:59.190700    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:59.407027 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:59.406931    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:59.789355 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:59.789315    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:59.986854 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:55:59.986813    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:00.182834 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:00.182802    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:00.237630 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:00.237602    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:02.478633 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:02.478604    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:02.576322 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:02.576285    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:02.637265 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:02.637229    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:02.994617 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:02.994586    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:03.140519 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:03.140484    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:03.211389 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:03.211359    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:03.348447 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:03.348356    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:03.462799 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:03.462761    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:03.582520 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:03.582487    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:03.639293 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:03.639198    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:03.965249 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:03.965216    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:04.155767 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:04.155736    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:04.254387 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:04.254300    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:04.587652 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:04.587571    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:04.703759 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:04.703730    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:05.379325 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:05.379287    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:05.434067 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:05.434034    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:10.376993 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:10.376954    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:10.431365 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:10.431337    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:14.173592 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:14.173557    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:14.225103 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:14.225070    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:15.012554 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:15.012503    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:56:15.014784 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:15.014756    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3337cbb2-24ee-40cd-8934-2f11d0069e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48skw\" (UID: \"3337cbb2-24ee-40cd-8934-2f11d0069e0f\") " pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:56:15.294953 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:15.294854    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-48skw"
Sep 09 15:56:15.294953 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:56:15.294906    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hdkxg" podUID="6833917f-b72a-4586-98ce-c68895a682f1"
Sep 09 15:56:15.561357 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:15.561285    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-48skw"]
Sep 09 15:56:15.564234 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:56:15.564211    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3337cbb2_24ee_40cd_8934_2f11d0069e0f.slice/crio-dcad9fa15f7e43bb081153b86bd8b689a0f21ba6d1c298f86fd3462e7d613ffc WatchSource:0}: Error finding container dcad9fa15f7e43bb081153b86bd8b689a0f21ba6d1c298f86fd3462e7d613ffc: Status 404 returned error can't find the container with id dcad9fa15f7e43bb081153b86bd8b689a0f21ba6d1c298f86fd3462e7d613ffc
Sep 09 15:56:15.657991 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:15.657962    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48skw" event={"ID":"3337cbb2-24ee-40cd-8934-2f11d0069e0f","Type":"ContainerStarted","Data":"3076610ad6eafe4759f9c6e0f02c356102e01dd73506ebccb79030a43d541022"}
Sep 09 15:56:15.657991 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:15.657991    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:56:15.658167 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:15.658015    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48skw" event={"ID":"3337cbb2-24ee-40cd-8934-2f11d0069e0f","Type":"ContainerStarted","Data":"dcad9fa15f7e43bb081153b86bd8b689a0f21ba6d1c298f86fd3462e7d613ffc"}
Sep 09 15:56:16.662164 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:16.662086    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48skw" event={"ID":"3337cbb2-24ee-40cd-8934-2f11d0069e0f","Type":"ContainerStarted","Data":"7bd5dc8cf6320f36ca85f434a962c71f81e6d62009e9865fcafbf6add1b3e0a3"}
Sep 09 15:56:18.437098 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:18.437057    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:56:18.439240 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:18.439222    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6833917f-b72a-4586-98ce-c68895a682f1-metrics-tls\") pod \"dns-default-hdkxg\" (UID: \"6833917f-b72a-4586-98ce-c68895a682f1\") " pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:56:18.661633 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:18.661602    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5xfw9\""
Sep 09 15:56:18.669317 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:18.669297    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:56:18.669761 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:18.669742    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48skw" event={"ID":"3337cbb2-24ee-40cd-8934-2f11d0069e0f","Type":"ContainerStarted","Data":"43d5cf258262a16816d337e0251637a2fd71bd1423cd5d0a0e5c58dd3a48ce03"}
Sep 09 15:56:18.687282 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:18.687209    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-48skw" podStartSLOduration=33.281713695 podStartE2EDuration="35.687195047s" totalImagesPullingTime="2.405481352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:55:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:18.686120248 +0000 UTC m=+283.552631471" watchObservedRunningTime="2026-09-09 15:56:18.687195047 +0000 UTC m=+283.553706266"
Sep 09 15:56:18.936163 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:18.936118    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hdkxg"]
Sep 09 15:56:18.938834 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:56:18.938765    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6833917f_b72a_4586_98ce_c68895a682f1.slice/crio-56f1ad763db361aef67af387ae984b64a528db40b954ddad02b9a35b75aebd9a WatchSource:0}: Error finding container 56f1ad763db361aef67af387ae984b64a528db40b954ddad02b9a35b75aebd9a: Status 404 returned error can't find the container with id 56f1ad763db361aef67af387ae984b64a528db40b954ddad02b9a35b75aebd9a
Sep 09 15:56:19.674337 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:19.674300    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hdkxg" event={"ID":"6833917f-b72a-4586-98ce-c68895a682f1","Type":"ContainerStarted","Data":"56f1ad763db361aef67af387ae984b64a528db40b954ddad02b9a35b75aebd9a"}
Sep 09 15:56:21.427954 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:21.427919    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:21.482418 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:21.482388    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:21.574609 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:21.574570    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:21.682229 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:21.682154    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hdkxg" event={"ID":"6833917f-b72a-4586-98ce-c68895a682f1","Type":"ContainerStarted","Data":"938166b552d2bab66f4163a9cc0c86161841b40fb61d7bc4abf970ce99d36588"}
Sep 09 15:56:21.682229 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:21.682203    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hdkxg" event={"ID":"6833917f-b72a-4586-98ce-c68895a682f1","Type":"ContainerStarted","Data":"6bfc9736796460c1fcb342cb7b8a9d38f99c922b8fd786d806e56918652f03f4"}
Sep 09 15:56:21.682399 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:21.682295    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:56:21.698383 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:21.698340    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hdkxg" podStartSLOduration=251.919194978 podStartE2EDuration="4m13.698326517s" totalImagesPullingTime="1.779131539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:21.69788 +0000 UTC m=+286.564391223" watchObservedRunningTime="2026-09-09 15:56:21.698326517 +0000 UTC m=+286.564837770"
Sep 09 15:56:25.758904 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:56:25.758866    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4tmj4" podUID="72246d50-b16d-4f50-b064-aa2a85cf2970"
Sep 09 15:56:26.946965 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:26.946932    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:29.569559 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:29.569520    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:30.222275 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:30.222235    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:30.283680 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:30.283650    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:30.688705 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:30.688673    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:31.215212 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:31.215174    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:31.686902 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:31.686876    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hdkxg"
Sep 09 15:56:32.416723 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:32.416691    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:32.616066 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:32.616011    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:32.671957 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:32.671873    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:32.872247 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:32.872210    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:32.973224 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:32.973187    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:33.152771 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.152742    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:33.250569 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.250448    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:33.407345 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.407311    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:33.462924 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.462894    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zg4bq"]
Sep 09 15:56:33.466055 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.466035    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"]
Sep 09 15:56:33.466192 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.466171    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.468756 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.468733    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 15:56:33.468881 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.468820    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 15:56:33.468959 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.468885    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 15:56:33.468959 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.468746    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 15:56:33.469065 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.469022    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 15:56:33.469132 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.469115    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 15:56:33.469363 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.469343    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pqjbj\""
Sep 09 15:56:33.469458 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.469386    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.471254 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.471235    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 09 15:56:33.471687 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.471559    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 09 15:56:33.471687 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.471575    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 09 15:56:33.471687 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.471590    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ckmsc\""
Sep 09 15:56:33.477079 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.477054    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"]
Sep 09 15:56:33.570091 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.569988    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:33.644370 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.644327    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.644573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.644374    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-wtmp\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.644573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.644426    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.644573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.644484    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.644573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.644504    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-tls\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.644573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.644524    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8591e18a-a4e5-489c-a3d2-d914123c9bd3-sys\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.644573 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.644546    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8591e18a-a4e5-489c-a3d2-d914123c9bd3-metrics-client-ca\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.644881 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.644575    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-accelerators-collector-config\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.644881 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.644680    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-textfile\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.644881 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.644723    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.644881 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.644752    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.644881 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.644776    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8591e18a-a4e5-489c-a3d2-d914123c9bd3-root\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.644881 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.644807    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s4m5\" (UniqueName: \"kubernetes.io/projected/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-kube-api-access-5s4m5\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.644881 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.644852    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n9rh\" (UniqueName: \"kubernetes.io/projected/8591e18a-a4e5-489c-a3d2-d914123c9bd3-kube-api-access-5n9rh\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.645208 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.644888    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.736731 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.736697    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:33.745774 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.745742    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n9rh\" (UniqueName: \"kubernetes.io/projected/8591e18a-a4e5-489c-a3d2-d914123c9bd3-kube-api-access-5n9rh\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.745924 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.745791    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.745924 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.745842    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.745924 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.745866    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-wtmp\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.745924 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.745896    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.746131 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.745933    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.746131 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:56:33.745957    2556 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 09 15:56:33.746131 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:56:33.746033    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-kube-state-metrics-tls podName:03edb11a-1a2d-49ea-aacf-3b6c4318a08f nodeName:}" failed. No retries permitted until 2026-09-09 15:56:34.246014326 +0000 UTC m=+299.112525527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-w2j7v" (UID: "03edb11a-1a2d-49ea-aacf-3b6c4318a08f") : secret "kube-state-metrics-tls" not found
Sep 09 15:56:33.746131 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:56:33.746039    2556 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 15:56:33.746131 ip-10-0-133-93 kubenswrapper[2556]: E0909 15:56:33.746086    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-tls podName:8591e18a-a4e5-489c-a3d2-d914123c9bd3 nodeName:}" failed. No retries permitted until 2026-09-09 15:56:34.246070367 +0000 UTC m=+299.112581572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-tls") pod "node-exporter-zg4bq" (UID: "8591e18a-a4e5-489c-a3d2-d914123c9bd3") : secret "node-exporter-tls" not found
Sep 09 15:56:33.746131 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.745959    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-tls\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.746131 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.746113    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8591e18a-a4e5-489c-a3d2-d914123c9bd3-sys\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.746504 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.746144    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8591e18a-a4e5-489c-a3d2-d914123c9bd3-metrics-client-ca\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.746504 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.746176    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-accelerators-collector-config\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.746504 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.746036    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-wtmp\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.746504 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.746212    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-textfile\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.746504 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.746246    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.746504 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.746275    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.746504 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.746299    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8591e18a-a4e5-489c-a3d2-d914123c9bd3-root\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.746504 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.746329    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s4m5\" (UniqueName: \"kubernetes.io/projected/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-kube-api-access-5s4m5\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.746504 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.746352    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.746940 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.746742    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.746940 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.746778    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8591e18a-a4e5-489c-a3d2-d914123c9bd3-sys\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.746940 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.746778    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8591e18a-a4e5-489c-a3d2-d914123c9bd3-metrics-client-ca\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.746940 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.746778    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.746940 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.746803    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8591e18a-a4e5-489c-a3d2-d914123c9bd3-root\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.747178 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.747009    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-textfile\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.747230 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.747190    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-accelerators-collector-config\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.749369 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.749344    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.749509 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.749490    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.755951 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.755929    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s4m5\" (UniqueName: \"kubernetes.io/projected/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-kube-api-access-5s4m5\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:33.756259 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.756241    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n9rh\" (UniqueName: \"kubernetes.io/projected/8591e18a-a4e5-489c-a3d2-d914123c9bd3-kube-api-access-5n9rh\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:33.835868 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.835781    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:33.936832 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:33.936797    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:34.038829 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:34.038798    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:34.250798 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:34.250770    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-tls\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:34.251003 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:34.250843    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:34.253247 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:34.253221    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8591e18a-a4e5-489c-a3d2-d914123c9bd3-node-exporter-tls\") pod \"node-exporter-zg4bq\" (UID: \"8591e18a-a4e5-489c-a3d2-d914123c9bd3\") " pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:34.253352 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:34.253260    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/03edb11a-1a2d-49ea-aacf-3b6c4318a08f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-w2j7v\" (UID: \"03edb11a-1a2d-49ea-aacf-3b6c4318a08f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:34.379953 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:34.379920    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zg4bq"
Sep 09 15:56:34.388072 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:34.388050    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"
Sep 09 15:56:34.391272 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:56:34.391247    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8591e18a_a4e5_489c_a3d2_d914123c9bd3.slice/crio-3c9d15fb3adb4a12aef36bd19a1baaa3dc2608dfef3bd1f7ffba0627233c166e WatchSource:0}: Error finding container 3c9d15fb3adb4a12aef36bd19a1baaa3dc2608dfef3bd1f7ffba0627233c166e: Status 404 returned error can't find the container with id 3c9d15fb3adb4a12aef36bd19a1baaa3dc2608dfef3bd1f7ffba0627233c166e
Sep 09 15:56:34.437000 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:34.436936    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:34.543078 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:34.542988    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:34.665826 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:34.665787    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v"]
Sep 09 15:56:34.669067 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:56:34.669035    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03edb11a_1a2d_49ea_aacf_3b6c4318a08f.slice/crio-044cfede2cb5301994f7750585ed1dd1c216a3849f23c2c9061ea50dcf184b71 WatchSource:0}: Error finding container 044cfede2cb5301994f7750585ed1dd1c216a3849f23c2c9061ea50dcf184b71: Status 404 returned error can't find the container with id 044cfede2cb5301994f7750585ed1dd1c216a3849f23c2c9061ea50dcf184b71
Sep 09 15:56:34.694177 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:34.694152    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:34.725405 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:34.725375    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v" event={"ID":"03edb11a-1a2d-49ea-aacf-3b6c4318a08f","Type":"ContainerStarted","Data":"044cfede2cb5301994f7750585ed1dd1c216a3849f23c2c9061ea50dcf184b71"}
Sep 09 15:56:34.726498 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:34.726448    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zg4bq" event={"ID":"8591e18a-a4e5-489c-a3d2-d914123c9bd3","Type":"ContainerStarted","Data":"3c9d15fb3adb4a12aef36bd19a1baaa3dc2608dfef3bd1f7ffba0627233c166e"}
Sep 09 15:56:35.568528 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:35.568495    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 15:56:35.569778 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:35.569741    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 15:56:35.575590 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:35.575540    2556 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 15:56:36.364838 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:36.364750    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:36.425638 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:36.425595    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:36.540624 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:36.540587    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:36.736599 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:36.736558    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v" event={"ID":"03edb11a-1a2d-49ea-aacf-3b6c4318a08f","Type":"ContainerStarted","Data":"89371af5cb695ece7fd03f9e7d5a1841427caee485f86d7c09018bf5e9aff92b"}
Sep 09 15:56:36.737086 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:36.736611    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v" event={"ID":"03edb11a-1a2d-49ea-aacf-3b6c4318a08f","Type":"ContainerStarted","Data":"23490205fe9ced3fa18dbde04e1311b3e3c28d94d0bc525deef74f999617e67c"}
Sep 09 15:56:36.737086 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:36.736631    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v" event={"ID":"03edb11a-1a2d-49ea-aacf-3b6c4318a08f","Type":"ContainerStarted","Data":"2a473e7980bf482a7d63915fc7c17fc92a8635db10a27879ed8ec8d740421f2d"}
Sep 09 15:56:36.754123 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:36.754075    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-w2j7v" podStartSLOduration=2.430986118 podStartE2EDuration="3.754057731s" totalImagesPullingTime="1.323071613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:36.75282312 +0000 UTC m=+301.619334339" watchObservedRunningTime="2026-09-09 15:56:36.754057731 +0000 UTC m=+301.620568952"
Sep 09 15:56:36.757741 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:36.757723    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:56:36.872213 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:36.872176    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:38.261239 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:38.261196    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:38.324061 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:38.324026    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:38.426384 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:38.426341    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:39.751433 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:39.751400    2556 generic.go:356] "Generic (PLEG): container finished" podID="8591e18a-a4e5-489c-a3d2-d914123c9bd3" containerID="76de31aaeb90b149f0c29ede4786698fb06bd829dacc1cf8420b59c6611001a0" exitCode=0
Sep 09 15:56:39.751879 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:39.751500    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zg4bq" event={"ID":"8591e18a-a4e5-489c-a3d2-d914123c9bd3","Type":"ContainerDied","Data":"76de31aaeb90b149f0c29ede4786698fb06bd829dacc1cf8420b59c6611001a0"}
Sep 09 15:56:40.757081 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:40.757039    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zg4bq" event={"ID":"8591e18a-a4e5-489c-a3d2-d914123c9bd3","Type":"ContainerStarted","Data":"9700a9e433c840e681956a9b518852bb0d4511856e00c6fad84a65734f17ed98"}
Sep 09 15:56:40.757590 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:40.757110    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zg4bq" event={"ID":"8591e18a-a4e5-489c-a3d2-d914123c9bd3","Type":"ContainerStarted","Data":"d791348f5cf12d259c55eadb16bc5c02039e08eb92de68bf87cc957cecd6ae32"}
Sep 09 15:56:40.776248 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:40.776188    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zg4bq" podStartSLOduration=3.266839417 podStartE2EDuration="7.776172569s" totalImagesPullingTime="4.449861619s" totalInitContainerRuntime="59.471533ms" isStatefulPod=true podCreationTimestamp="2026-09-09 15:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:40.774320079 +0000 UTC m=+305.640831298" watchObservedRunningTime="2026-09-09 15:56:40.776172569 +0000 UTC m=+305.642683789"
Sep 09 15:56:41.541539 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:41.541497    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:41.642444 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:41.642406    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:42.121584 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:42.121538    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:42.585047 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:42.585014    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:45.188926 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:45.188892    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:48.497239 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:48.497195    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:49.156374 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:49.156334    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:49.208419 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:49.208386    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:49.503510 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:49.503455    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:49.604049 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:49.604014    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:49.705394 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:56:49.705361    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:00.198499 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:00.198434    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:00.268931 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:00.268901    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:09.098810 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:09.098763    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln" podUID="47278756-4b7f-42be-861b-5faf5c764a30" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 15:57:19.098295 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:19.098257    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln" podUID="47278756-4b7f-42be-861b-5faf5c764a30" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 15:57:29.098575 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:29.098492    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln" podUID="47278756-4b7f-42be-861b-5faf5c764a30" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 15:57:29.098575 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:29.098564    2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln"
Sep 09 15:57:29.099065 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:29.099036    2556 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"fe41aa4de4b6ea05fb82894b5dc4047c01eacf755fbf2016628faa3dd3dd10b5"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 09 15:57:29.099107 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:29.099086    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln" podUID="47278756-4b7f-42be-861b-5faf5c764a30" containerName="service-proxy" containerID="cri-o://fe41aa4de4b6ea05fb82894b5dc4047c01eacf755fbf2016628faa3dd3dd10b5" gracePeriod=30
Sep 09 15:57:29.911574 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:29.911540    2556 generic.go:356] "Generic (PLEG): container finished" podID="47278756-4b7f-42be-861b-5faf5c764a30" containerID="fe41aa4de4b6ea05fb82894b5dc4047c01eacf755fbf2016628faa3dd3dd10b5" exitCode=2
Sep 09 15:57:29.911574 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:29.911584    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln" event={"ID":"47278756-4b7f-42be-861b-5faf5c764a30","Type":"ContainerDied","Data":"fe41aa4de4b6ea05fb82894b5dc4047c01eacf755fbf2016628faa3dd3dd10b5"}
Sep 09 15:57:29.911780 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:29.911614    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7694b94fc9-xmqln" event={"ID":"47278756-4b7f-42be-861b-5faf5c764a30","Type":"ContainerStarted","Data":"2f765a6cf6781b304b3281963e75a5bf12e9da71e11a91f622f46203c67184e7"}
Sep 09 15:57:38.554222 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:38.554188    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:38.616706 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:38.616676    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:45.663641 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:45.663607    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:45.765225 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:45.765195    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:49.504114 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:49.504073    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:57:49.506346 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:49.506322    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72246d50-b16d-4f50-b064-aa2a85cf2970-metrics-certs\") pod \"network-metrics-daemon-4tmj4\" (UID: \"72246d50-b16d-4f50-b064-aa2a85cf2970\") " pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:57:49.661152 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:49.661119    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f2dhs\""
Sep 09 15:57:49.668884 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:49.668866    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tmj4"
Sep 09 15:57:49.935717 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:49.935542    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4tmj4"]
Sep 09 15:57:49.938041 ip-10-0-133-93 kubenswrapper[2556]: W0909 15:57:49.938015    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72246d50_b16d_4f50_b064_aa2a85cf2970.slice/crio-d0f2b4eef8f7d7d3257b35b10ecaf1a235d860ba11b3e01720b6cb9a2172af5c WatchSource:0}: Error finding container d0f2b4eef8f7d7d3257b35b10ecaf1a235d860ba11b3e01720b6cb9a2172af5c: Status 404 returned error can't find the container with id d0f2b4eef8f7d7d3257b35b10ecaf1a235d860ba11b3e01720b6cb9a2172af5c
Sep 09 15:57:49.939942 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:49.939928    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 15:57:49.978173 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:49.978145    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4tmj4" event={"ID":"72246d50-b16d-4f50-b064-aa2a85cf2970","Type":"ContainerStarted","Data":"d0f2b4eef8f7d7d3257b35b10ecaf1a235d860ba11b3e01720b6cb9a2172af5c"}
Sep 09 15:57:50.983635 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:50.983611    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4tmj4" event={"ID":"72246d50-b16d-4f50-b064-aa2a85cf2970","Type":"ContainerStarted","Data":"1da0896dd114aaa224aab0a82e65d8655755093e294dfa83ad90d771860c6fba"}
Sep 09 15:57:51.989012 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:51.988976    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4tmj4" event={"ID":"72246d50-b16d-4f50-b064-aa2a85cf2970","Type":"ContainerStarted","Data":"516fa02ad037c624a9ae14c4a2aa196200b1ce312e6786bddff2916919f33fea"}
Sep 09 15:57:52.006836 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:52.006786    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4tmj4" podStartSLOduration=376.051717425 podStartE2EDuration="6m17.006770416s" totalImagesPullingTime="955.052991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:57:52.005684911 +0000 UTC m=+376.872196131" watchObservedRunningTime="2026-09-09 15:57:52.006770416 +0000 UTC m=+376.873281636"
Sep 09 15:57:52.112527 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:52.112493    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:52.213606 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:52.213573    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:52.620120 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:52.620091    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:52.752049 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:57:52.752007    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:01.706523 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:01.706478    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:01.806428 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:01.806389    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:07.851850 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:07.851811    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:07.907229 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:07.907198    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:12.827434 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:12.827394    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:12.885992 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:12.885960    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:13.154735 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:13.154646    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:19.743936 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:19.743906    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:19.797120 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:19.797084    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:23.856411 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:23.856368    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:23.907200 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:23.907162    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:23.959710 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:23.959659    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:30.198658 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:30.198618    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:30.299162 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:30.299116    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:39.407577 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:39.407502    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:39.461151 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:39.461114    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:56.683384 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:58:56.683351    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 15:59:35.002410 ip-10-0-133-93 kubenswrapper[2556]: I0909 15:59:35.002372    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:08.725057 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:08.725024    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:09.542606 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:09.542575    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:14.602794 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:14.602741    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:14.659813 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:14.659772    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:25.395801 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:25.395767    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:35.406951 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:35.406863    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:36.194769 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:36.194691    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:39.011603 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:39.011566    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:39.113761 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:39.113711    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:42.181958 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:42.181925    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:43.475912 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:43.475882    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:43.576404 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:43.576367    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:44.391645 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:44.391605    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:44.582714 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:44.582682    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:44.639968 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:44.639935    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:46.653563 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:46.653533    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:46.722283 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:46.722244    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:47.898842 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:47.898803    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:52.790031 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:52.789994    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:55.027010 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:55.026977    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:55.085605 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:55.085570    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:56.143064 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:56.143027    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sglz8"]
Sep 09 16:00:56.146026 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:56.146003    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sglz8"
Sep 09 16:00:56.148282 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:56.148256    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-sg68w\""
Sep 09 16:00:56.148282 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:56.148276    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 16:00:56.148555 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:56.148258    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 16:00:56.155691 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:56.155670    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sglz8"]
Sep 09 16:00:56.188950 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:56.188926    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fa70f62-3193-4fdb-872d-a7ec355d18b9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-sglz8\" (UID: \"1fa70f62-3193-4fdb-872d-a7ec355d18b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sglz8"
Sep 09 16:00:56.189104 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:56.188956    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4blxd\" (UniqueName: \"kubernetes.io/projected/1fa70f62-3193-4fdb-872d-a7ec355d18b9-kube-api-access-4blxd\") pod \"openshift-lws-operator-fd8ccff4c-sglz8\" (UID: \"1fa70f62-3193-4fdb-872d-a7ec355d18b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sglz8"
Sep 09 16:00:56.289275 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:56.289241    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fa70f62-3193-4fdb-872d-a7ec355d18b9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-sglz8\" (UID: \"1fa70f62-3193-4fdb-872d-a7ec355d18b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sglz8"
Sep 09 16:00:56.289275 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:56.289277    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4blxd\" (UniqueName: \"kubernetes.io/projected/1fa70f62-3193-4fdb-872d-a7ec355d18b9-kube-api-access-4blxd\") pod \"openshift-lws-operator-fd8ccff4c-sglz8\" (UID: \"1fa70f62-3193-4fdb-872d-a7ec355d18b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sglz8"
Sep 09 16:00:56.289700 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:56.289676    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fa70f62-3193-4fdb-872d-a7ec355d18b9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-sglz8\" (UID: \"1fa70f62-3193-4fdb-872d-a7ec355d18b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sglz8"
Sep 09 16:00:56.296983 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:56.296956    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4blxd\" (UniqueName: \"kubernetes.io/projected/1fa70f62-3193-4fdb-872d-a7ec355d18b9-kube-api-access-4blxd\") pod \"openshift-lws-operator-fd8ccff4c-sglz8\" (UID: \"1fa70f62-3193-4fdb-872d-a7ec355d18b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sglz8"
Sep 09 16:00:56.459287 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:56.459252    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sglz8"
Sep 09 16:00:56.732430 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:56.732354    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sglz8"]
Sep 09 16:00:56.735608 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:00:56.735575    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fa70f62_3193_4fdb_872d_a7ec355d18b9.slice/crio-5936cbd866cfd52d8811004b0aa2bdd6e4867edfb6e0fc53584b5d7b81998a70 WatchSource:0}: Error finding container 5936cbd866cfd52d8811004b0aa2bdd6e4867edfb6e0fc53584b5d7b81998a70: Status 404 returned error can't find the container with id 5936cbd866cfd52d8811004b0aa2bdd6e4867edfb6e0fc53584b5d7b81998a70
Sep 09 16:00:57.597841 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:57.597800    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sglz8" event={"ID":"1fa70f62-3193-4fdb-872d-a7ec355d18b9","Type":"ContainerStarted","Data":"5936cbd866cfd52d8811004b0aa2bdd6e4867edfb6e0fc53584b5d7b81998a70"}
Sep 09 16:00:59.606392 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:59.606303    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sglz8" event={"ID":"1fa70f62-3193-4fdb-872d-a7ec355d18b9","Type":"ContainerStarted","Data":"0b9b9071eb0a2f1f040ab70c888a0585c1d0b1f78ca32749c01aa7020291c555"}
Sep 09 16:00:59.621247 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:00:59.621191    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sglz8" podStartSLOduration=1.05841291 podStartE2EDuration="3.621172935s" totalImagesPullingTime="2.562760025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:00:59.620640905 +0000 UTC m=+564.487152124" watchObservedRunningTime="2026-09-09 16:00:59.621172935 +0000 UTC m=+564.487684154"
Sep 09 16:01:00.267197 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:00.267159    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:00.368055 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:00.368019    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:02.791571 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:02.791526    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:18.285078 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:18.285036    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:18.389148 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:18.389105    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:18.545518 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:18.545411    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:18.651400 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:18.651357    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:18.842236 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:18.842136    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:19.009802 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.009752    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:19.107949 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.107855    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:19.151425 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.151394    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"]
Sep 09 16:01:19.154588 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.154567    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.157801 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.157773    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 16:01:19.158552 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.158533    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jndt2\""
Sep 09 16:01:19.158846 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.158819    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 16:01:19.159044 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.159030    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 16:01:19.159115 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.159052    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 16:01:19.195354 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.195325    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"]
Sep 09 16:01:19.254141 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.254106    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efca11da-bf8c-469c-b9e1-904559b5c395-webhook-cert\") pod \"opendatahub-operator-controller-manager-b4ccd886c-lfmxt\" (UID: \"efca11da-bf8c-469c-b9e1-904559b5c395\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.254141 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.254140    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzkqg\" (UniqueName: \"kubernetes.io/projected/efca11da-bf8c-469c-b9e1-904559b5c395-kube-api-access-bzkqg\") pod \"opendatahub-operator-controller-manager-b4ccd886c-lfmxt\" (UID: \"efca11da-bf8c-469c-b9e1-904559b5c395\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.254388 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.254164    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/efca11da-bf8c-469c-b9e1-904559b5c395-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b4ccd886c-lfmxt\" (UID: \"efca11da-bf8c-469c-b9e1-904559b5c395\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.254388 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.254181    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efca11da-bf8c-469c-b9e1-904559b5c395-tmp\") pod \"opendatahub-operator-controller-manager-b4ccd886c-lfmxt\" (UID: \"efca11da-bf8c-469c-b9e1-904559b5c395\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.254388 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.254276    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/efca11da-bf8c-469c-b9e1-904559b5c395-manifests\") pod \"opendatahub-operator-controller-manager-b4ccd886c-lfmxt\" (UID: \"efca11da-bf8c-469c-b9e1-904559b5c395\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.331773 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.331734    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:19.354696 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.354663    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efca11da-bf8c-469c-b9e1-904559b5c395-webhook-cert\") pod \"opendatahub-operator-controller-manager-b4ccd886c-lfmxt\" (UID: \"efca11da-bf8c-469c-b9e1-904559b5c395\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.354696 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.354700    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzkqg\" (UniqueName: \"kubernetes.io/projected/efca11da-bf8c-469c-b9e1-904559b5c395-kube-api-access-bzkqg\") pod \"opendatahub-operator-controller-manager-b4ccd886c-lfmxt\" (UID: \"efca11da-bf8c-469c-b9e1-904559b5c395\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.354906 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.354724    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/efca11da-bf8c-469c-b9e1-904559b5c395-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b4ccd886c-lfmxt\" (UID: \"efca11da-bf8c-469c-b9e1-904559b5c395\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.354906 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.354802    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efca11da-bf8c-469c-b9e1-904559b5c395-tmp\") pod \"opendatahub-operator-controller-manager-b4ccd886c-lfmxt\" (UID: \"efca11da-bf8c-469c-b9e1-904559b5c395\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.354906 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.354876    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/efca11da-bf8c-469c-b9e1-904559b5c395-manifests\") pod \"opendatahub-operator-controller-manager-b4ccd886c-lfmxt\" (UID: \"efca11da-bf8c-469c-b9e1-904559b5c395\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.355304 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.355281    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/efca11da-bf8c-469c-b9e1-904559b5c395-manifests\") pod \"opendatahub-operator-controller-manager-b4ccd886c-lfmxt\" (UID: \"efca11da-bf8c-469c-b9e1-904559b5c395\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.357234 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.357205    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efca11da-bf8c-469c-b9e1-904559b5c395-tmp\") pod \"opendatahub-operator-controller-manager-b4ccd886c-lfmxt\" (UID: \"efca11da-bf8c-469c-b9e1-904559b5c395\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.357346 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.357301    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/efca11da-bf8c-469c-b9e1-904559b5c395-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b4ccd886c-lfmxt\" (UID: \"efca11da-bf8c-469c-b9e1-904559b5c395\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.357346 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.357322    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efca11da-bf8c-469c-b9e1-904559b5c395-webhook-cert\") pod \"opendatahub-operator-controller-manager-b4ccd886c-lfmxt\" (UID: \"efca11da-bf8c-469c-b9e1-904559b5c395\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.363111 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.363040    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzkqg\" (UniqueName: \"kubernetes.io/projected/efca11da-bf8c-469c-b9e1-904559b5c395-kube-api-access-bzkqg\") pod \"opendatahub-operator-controller-manager-b4ccd886c-lfmxt\" (UID: \"efca11da-bf8c-469c-b9e1-904559b5c395\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.432186 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.432153    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:19.464854 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.464822    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:19.532637 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.532605    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:19.633069 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.632984    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:19.732844 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.732811    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:19.746313 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.746274    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"]
Sep 09 16:01:19.750823 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:01:19.750797    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefca11da_bf8c_469c_b9e1_904559b5c395.slice/crio-0f0f2629000020702c316073c383c8a9f81f369c88454d42c22918fe399b2215 WatchSource:0}: Error finding container 0f0f2629000020702c316073c383c8a9f81f369c88454d42c22918fe399b2215: Status 404 returned error can't find the container with id 0f0f2629000020702c316073c383c8a9f81f369c88454d42c22918fe399b2215
Sep 09 16:01:19.841828 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.841792    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:19.933711 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:19.933673    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:20.681345 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:20.681310    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt" event={"ID":"efca11da-bf8c-469c-b9e1-904559b5c395","Type":"ContainerStarted","Data":"0f0f2629000020702c316073c383c8a9f81f369c88454d42c22918fe399b2215"}
Sep 09 16:01:23.699444 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:23.699409    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt" event={"ID":"efca11da-bf8c-469c-b9e1-904559b5c395","Type":"ContainerStarted","Data":"cc21931c02a1c3d67682a50caddb4776c41bc6ad8f950fe70544d1745e2403dc"}
Sep 09 16:01:24.704742 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:24.704706    2556 generic.go:356] "Generic (PLEG): container finished" podID="efca11da-bf8c-469c-b9e1-904559b5c395" containerID="cc21931c02a1c3d67682a50caddb4776c41bc6ad8f950fe70544d1745e2403dc" exitCode=0
Sep 09 16:01:24.705142 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:24.704755    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt" event={"ID":"efca11da-bf8c-469c-b9e1-904559b5c395","Type":"ContainerDied","Data":"cc21931c02a1c3d67682a50caddb4776c41bc6ad8f950fe70544d1745e2403dc"}
Sep 09 16:01:24.789271 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:24.789228    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:24.889936 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:24.889901    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:25.709857 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:25.709818    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt" event={"ID":"efca11da-bf8c-469c-b9e1-904559b5c395","Type":"ContainerStarted","Data":"05021b5d3f83901d664a01a8cd71b9cd06b2cd96ef4e43b169b7277b86aa5b31"}
Sep 09 16:01:25.710336 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:25.709925    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:25.731232 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:25.731171    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt" podStartSLOduration=2.009150527 podStartE2EDuration="6.731152806s" totalImagesPullingTime="3.673729046s" totalInitContainerRuntime="1.048273233s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:01:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:01:25.729800487 +0000 UTC m=+590.596311706" watchObservedRunningTime="2026-09-09 16:01:25.731152806 +0000 UTC m=+590.597664025"
Sep 09 16:01:29.784486 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:29.784433    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:31.604927 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:31.604883    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:35.592194 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:35.592166    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 16:01:35.592661 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:35.592304    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 16:01:36.716067 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:36.716035    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-lfmxt"
Sep 09 16:01:40.352765 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:40.352731    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:40.561074 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:40.561041    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:42.234994 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.234890    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:42.269954 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.269918    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-747b697884-whtvt"]
Sep 09 16:01:42.271894 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.271879    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-747b697884-whtvt"
Sep 09 16:01:42.275203 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.275174    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 16:01:42.275203 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.275198    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 16:01:42.275383 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.275173    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 16:01:42.275767 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.275751    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 16:01:42.286410 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.286381    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-747b697884-whtvt"]
Sep 09 16:01:42.432992 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.432955    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4972d020-567d-464e-8f08-21ca7620cab0-tmp\") pod \"kube-auth-proxy-747b697884-whtvt\" (UID: \"4972d020-567d-464e-8f08-21ca7620cab0\") " pod="openshift-ingress/kube-auth-proxy-747b697884-whtvt"
Sep 09 16:01:42.433137 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.433013    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4972d020-567d-464e-8f08-21ca7620cab0-tls-certs\") pod \"kube-auth-proxy-747b697884-whtvt\" (UID: \"4972d020-567d-464e-8f08-21ca7620cab0\") " pod="openshift-ingress/kube-auth-proxy-747b697884-whtvt"
Sep 09 16:01:42.433137 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.433102    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwkxg\" (UniqueName: \"kubernetes.io/projected/4972d020-567d-464e-8f08-21ca7620cab0-kube-api-access-gwkxg\") pod \"kube-auth-proxy-747b697884-whtvt\" (UID: \"4972d020-567d-464e-8f08-21ca7620cab0\") " pod="openshift-ingress/kube-auth-proxy-747b697884-whtvt"
Sep 09 16:01:42.533995 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.533906    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4972d020-567d-464e-8f08-21ca7620cab0-tmp\") pod \"kube-auth-proxy-747b697884-whtvt\" (UID: \"4972d020-567d-464e-8f08-21ca7620cab0\") " pod="openshift-ingress/kube-auth-proxy-747b697884-whtvt"
Sep 09 16:01:42.533995 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.533952    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4972d020-567d-464e-8f08-21ca7620cab0-tls-certs\") pod \"kube-auth-proxy-747b697884-whtvt\" (UID: \"4972d020-567d-464e-8f08-21ca7620cab0\") " pod="openshift-ingress/kube-auth-proxy-747b697884-whtvt"
Sep 09 16:01:42.533995 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.533989    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwkxg\" (UniqueName: \"kubernetes.io/projected/4972d020-567d-464e-8f08-21ca7620cab0-kube-api-access-gwkxg\") pod \"kube-auth-proxy-747b697884-whtvt\" (UID: \"4972d020-567d-464e-8f08-21ca7620cab0\") " pod="openshift-ingress/kube-auth-proxy-747b697884-whtvt"
Sep 09 16:01:42.534213 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:01:42.534090    2556 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 09 16:01:42.534213 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:01:42.534189    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4972d020-567d-464e-8f08-21ca7620cab0-tls-certs podName:4972d020-567d-464e-8f08-21ca7620cab0 nodeName:}" failed. No retries permitted until 2026-09-09 16:01:43.03416896 +0000 UTC m=+607.900680161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4972d020-567d-464e-8f08-21ca7620cab0-tls-certs") pod "kube-auth-proxy-747b697884-whtvt" (UID: "4972d020-567d-464e-8f08-21ca7620cab0") : secret "kube-auth-proxy-tls" not found
Sep 09 16:01:42.536226 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.536198    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4972d020-567d-464e-8f08-21ca7620cab0-tmp\") pod \"kube-auth-proxy-747b697884-whtvt\" (UID: \"4972d020-567d-464e-8f08-21ca7620cab0\") " pod="openshift-ingress/kube-auth-proxy-747b697884-whtvt"
Sep 09 16:01:42.545164 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.545128    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwkxg\" (UniqueName: \"kubernetes.io/projected/4972d020-567d-464e-8f08-21ca7620cab0-kube-api-access-gwkxg\") pod \"kube-auth-proxy-747b697884-whtvt\" (UID: \"4972d020-567d-464e-8f08-21ca7620cab0\") " pod="openshift-ingress/kube-auth-proxy-747b697884-whtvt"
Sep 09 16:01:42.583120 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:42.583090    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:43.039103 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.039064    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4972d020-567d-464e-8f08-21ca7620cab0-tls-certs\") pod \"kube-auth-proxy-747b697884-whtvt\" (UID: \"4972d020-567d-464e-8f08-21ca7620cab0\") " pod="openshift-ingress/kube-auth-proxy-747b697884-whtvt"
Sep 09 16:01:43.041565 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.041541    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4972d020-567d-464e-8f08-21ca7620cab0-tls-certs\") pod \"kube-auth-proxy-747b697884-whtvt\" (UID: \"4972d020-567d-464e-8f08-21ca7620cab0\") " pod="openshift-ingress/kube-auth-proxy-747b697884-whtvt"
Sep 09 16:01:43.181680 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.181642    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-747b697884-whtvt"
Sep 09 16:01:43.449179 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.449146    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-747b697884-whtvt"]
Sep 09 16:01:43.452956 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:01:43.452931    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4972d020_567d_464e_8f08_21ca7620cab0.slice/crio-f3f2c6365b279c8115b3c34ca81d508fe8ed9fbfe4cd5e5364db2c722fb7b9bd WatchSource:0}: Error finding container f3f2c6365b279c8115b3c34ca81d508fe8ed9fbfe4cd5e5364db2c722fb7b9bd: Status 404 returned error can't find the container with id f3f2c6365b279c8115b3c34ca81d508fe8ed9fbfe4cd5e5364db2c722fb7b9bd
Sep 09 16:01:43.701009 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.700938    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"]
Sep 09 16:01:43.703675 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.703653    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.705998 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.705975    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 16:01:43.706133 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.705975    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 16:01:43.706299 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.706275    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 16:01:43.706299 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.706280    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xctbn\""
Sep 09 16:01:43.706604 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.706590    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:43.706991 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.706973    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 16:01:43.718380 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.718354    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"]
Sep 09 16:01:43.779951 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.779917    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-747b697884-whtvt" event={"ID":"4972d020-567d-464e-8f08-21ca7620cab0","Type":"ContainerStarted","Data":"f3f2c6365b279c8115b3c34ca81d508fe8ed9fbfe4cd5e5364db2c722fb7b9bd"}
Sep 09 16:01:43.804424 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.804394    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:43.845190 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.845153    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.845190 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.845190    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.845411 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.845215    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.845411 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.845265    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn99t\" (UniqueName: \"kubernetes.io/projected/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-kube-api-access-kn99t\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.845411 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.845300    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.845411 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.845342    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.845594 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.845404    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.946560 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.946517    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.946560 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.946560    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.946760 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.946589    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.946760 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.946615    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn99t\" (UniqueName: \"kubernetes.io/projected/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-kube-api-access-kn99t\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.946839 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.946784    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.946892 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.946847    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.946945 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.946900    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.947567 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.947538    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.948986 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.948966    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.949185 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.949165    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.949225 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.949209    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.949225 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.949215    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.954420 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.954365    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:43.954584 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:43.954567    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn99t\" (UniqueName: \"kubernetes.io/projected/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-kube-api-access-kn99t\") pod \"istiod-openshift-gateway-7b7995d5cd-q7grb\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:44.012894 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:44.012855    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:44.288393 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:44.288152    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"]
Sep 09 16:01:44.290681 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:01:44.290650    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c18115_32a9_4bd6_8a72_50ca08dbc9e2.slice/crio-adce4ed505aee16e82c34af4ad055c2d9163fbc588b9ad5227855c21eef509df WatchSource:0}: Error finding container adce4ed505aee16e82c34af4ad055c2d9163fbc588b9ad5227855c21eef509df: Status 404 returned error can't find the container with id adce4ed505aee16e82c34af4ad055c2d9163fbc588b9ad5227855c21eef509df
Sep 09 16:01:44.638796 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:44.638701    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:44.784176 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:44.784139    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb" event={"ID":"81c18115-32a9-4bd6-8a72-50ca08dbc9e2","Type":"ContainerStarted","Data":"adce4ed505aee16e82c34af4ad055c2d9163fbc588b9ad5227855c21eef509df"}
Sep 09 16:01:45.725047 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:45.724919    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:46.196947 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:46.196908    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:48.353940 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:48.353899    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:01:48.354207 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:48.353974    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:01:48.802201 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:48.802167    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb" event={"ID":"81c18115-32a9-4bd6-8a72-50ca08dbc9e2","Type":"ContainerStarted","Data":"2515a111fc2f28e51479984568c7a55851710d2f658268ebe8e8c39df20c38d9"}
Sep 09 16:01:48.802385 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:48.802234    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:48.803766 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:48.803739    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-747b697884-whtvt" event={"ID":"4972d020-567d-464e-8f08-21ca7620cab0","Type":"ContainerStarted","Data":"cd8d7e4e18b2a8604f3839c897b3d756f692628b30db6ef295cc138ebb92567e"}
Sep 09 16:01:48.826658 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:48.826606    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb" podStartSLOduration=1.765550881 podStartE2EDuration="5.82658879s" totalImagesPullingTime="4.061037909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:01:48.824453832 +0000 UTC m=+613.690965062" watchObservedRunningTime="2026-09-09 16:01:48.82658879 +0000 UTC m=+613.693100003"
Sep 09 16:01:48.863542 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:48.863485    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-747b697884-whtvt" podStartSLOduration=1.95788583 podStartE2EDuration="6.863452047s" totalImagesPullingTime="4.905566217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:01:48.863075874 +0000 UTC m=+613.729587094" watchObservedRunningTime="2026-09-09 16:01:48.863452047 +0000 UTC m=+613.729963267"
Sep 09 16:01:49.456098 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:49.456056    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:49.809171 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:49.809092    2556 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-q7grb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 16:01:49.809171 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:49.809149    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb" podUID="81c18115-32a9-4bd6-8a72-50ca08dbc9e2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 16:01:52.808730 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:52.808700    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:01:53.083404 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:53.083326    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:01:53.183709 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:01:53.183676    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:02:00.754972 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.754886    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"]
Sep 09 16:02:00.757249 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.757227    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:00.759755 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.759733    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hs9zx\""
Sep 09 16:02:00.760612 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.760591    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 16:02:00.760731 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.760686    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 16:02:00.760800 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.760735    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 16:02:00.770396 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.770367    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"]
Sep 09 16:02:00.782062 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.782031    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mb88\" (UniqueName: \"kubernetes.io/projected/d0de56ca-b8ec-4087-ae6f-16e839dfcebb-kube-api-access-8mb88\") pod \"lws-controller-manager-678f94cf5f-gw5mr\" (UID: \"d0de56ca-b8ec-4087-ae6f-16e839dfcebb\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:00.782205 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.782084    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0de56ca-b8ec-4087-ae6f-16e839dfcebb-metrics-cert\") pod \"lws-controller-manager-678f94cf5f-gw5mr\" (UID: \"d0de56ca-b8ec-4087-ae6f-16e839dfcebb\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:00.782205 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.782142    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0de56ca-b8ec-4087-ae6f-16e839dfcebb-cert\") pod \"lws-controller-manager-678f94cf5f-gw5mr\" (UID: \"d0de56ca-b8ec-4087-ae6f-16e839dfcebb\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:00.782205 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.782178    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d0de56ca-b8ec-4087-ae6f-16e839dfcebb-manager-config\") pod \"lws-controller-manager-678f94cf5f-gw5mr\" (UID: \"d0de56ca-b8ec-4087-ae6f-16e839dfcebb\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:00.883076 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.883041    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mb88\" (UniqueName: \"kubernetes.io/projected/d0de56ca-b8ec-4087-ae6f-16e839dfcebb-kube-api-access-8mb88\") pod \"lws-controller-manager-678f94cf5f-gw5mr\" (UID: \"d0de56ca-b8ec-4087-ae6f-16e839dfcebb\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:00.883289 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.883097    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0de56ca-b8ec-4087-ae6f-16e839dfcebb-metrics-cert\") pod \"lws-controller-manager-678f94cf5f-gw5mr\" (UID: \"d0de56ca-b8ec-4087-ae6f-16e839dfcebb\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:00.883356 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.883338    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0de56ca-b8ec-4087-ae6f-16e839dfcebb-cert\") pod \"lws-controller-manager-678f94cf5f-gw5mr\" (UID: \"d0de56ca-b8ec-4087-ae6f-16e839dfcebb\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:00.883412 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.883358    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d0de56ca-b8ec-4087-ae6f-16e839dfcebb-manager-config\") pod \"lws-controller-manager-678f94cf5f-gw5mr\" (UID: \"d0de56ca-b8ec-4087-ae6f-16e839dfcebb\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:00.883966 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.883944    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d0de56ca-b8ec-4087-ae6f-16e839dfcebb-manager-config\") pod \"lws-controller-manager-678f94cf5f-gw5mr\" (UID: \"d0de56ca-b8ec-4087-ae6f-16e839dfcebb\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:00.885786 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.885765    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0de56ca-b8ec-4087-ae6f-16e839dfcebb-metrics-cert\") pod \"lws-controller-manager-678f94cf5f-gw5mr\" (UID: \"d0de56ca-b8ec-4087-ae6f-16e839dfcebb\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:00.885863 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.885845    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0de56ca-b8ec-4087-ae6f-16e839dfcebb-cert\") pod \"lws-controller-manager-678f94cf5f-gw5mr\" (UID: \"d0de56ca-b8ec-4087-ae6f-16e839dfcebb\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:00.894330 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:00.894304    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mb88\" (UniqueName: \"kubernetes.io/projected/d0de56ca-b8ec-4087-ae6f-16e839dfcebb-kube-api-access-8mb88\") pod \"lws-controller-manager-678f94cf5f-gw5mr\" (UID: \"d0de56ca-b8ec-4087-ae6f-16e839dfcebb\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:01.066701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:01.066607    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:01.358447 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:01.358365    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"]
Sep 09 16:02:01.361451 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:02:01.361421    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0de56ca_b8ec_4087_ae6f_16e839dfcebb.slice/crio-cd2e94ebe07cf8189c6b8fb83471324d189f781fb96e39b918185b3edcb18f95 WatchSource:0}: Error finding container cd2e94ebe07cf8189c6b8fb83471324d189f781fb96e39b918185b3edcb18f95: Status 404 returned error can't find the container with id cd2e94ebe07cf8189c6b8fb83471324d189f781fb96e39b918185b3edcb18f95
Sep 09 16:02:01.855129 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:01.855096    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr" event={"ID":"d0de56ca-b8ec-4087-ae6f-16e839dfcebb","Type":"ContainerStarted","Data":"cd2e94ebe07cf8189c6b8fb83471324d189f781fb96e39b918185b3edcb18f95"}
Sep 09 16:02:03.797223 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:03.797193    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"]
Sep 09 16:02:03.797654 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:03.797399    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb" podUID="81c18115-32a9-4bd6-8a72-50ca08dbc9e2" containerName="discovery" containerID="cri-o://2515a111fc2f28e51479984568c7a55851710d2f658268ebe8e8c39df20c38d9" gracePeriod=30
Sep 09 16:02:03.865439 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:03.865395    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr" event={"ID":"d0de56ca-b8ec-4087-ae6f-16e839dfcebb","Type":"ContainerStarted","Data":"6657eb34cb19e145baf0f6f329a96b8ba51cfb86f3590fb581b637b85b5604dd"}
Sep 09 16:02:03.865706 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:03.865531    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:03.886330 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:03.886274    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr" podStartSLOduration=2.337377236 podStartE2EDuration="3.886256816s" totalImagesPullingTime="1.54887958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:02:03.885880393 +0000 UTC m=+628.752391613" watchObservedRunningTime="2026-09-09 16:02:03.886256816 +0000 UTC m=+628.752768035"
Sep 09 16:02:04.041794 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.041768    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:02:04.112422 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.112323    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn99t\" (UniqueName: \"kubernetes.io/projected/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-kube-api-access-kn99t\") pod \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") "
Sep 09 16:02:04.112422 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.112373    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-csr-dns-cert\") pod \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") "
Sep 09 16:02:04.112422 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.112413    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-cacerts\") pod \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") "
Sep 09 16:02:04.112727 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.112491    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-token\") pod \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") "
Sep 09 16:02:04.112727 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.112545    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-csr-ca-configmap\") pod \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") "
Sep 09 16:02:04.112727 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.112571    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-local-certs\") pod \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") "
Sep 09 16:02:04.112727 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.112593    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-kubeconfig\") pod \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\" (UID: \"81c18115-32a9-4bd6-8a72-50ca08dbc9e2\") "
Sep 09 16:02:04.112998 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.112951    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "81c18115-32a9-4bd6-8a72-50ca08dbc9e2" (UID: "81c18115-32a9-4bd6-8a72-50ca08dbc9e2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:02:04.114750 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.114719    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "81c18115-32a9-4bd6-8a72-50ca08dbc9e2" (UID: "81c18115-32a9-4bd6-8a72-50ca08dbc9e2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:02:04.114885 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.114806    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-local-certs" (OuterVolumeSpecName: "local-certs") pod "81c18115-32a9-4bd6-8a72-50ca08dbc9e2" (UID: "81c18115-32a9-4bd6-8a72-50ca08dbc9e2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:02:04.115144 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.115118    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-cacerts" (OuterVolumeSpecName: "cacerts") pod "81c18115-32a9-4bd6-8a72-50ca08dbc9e2" (UID: "81c18115-32a9-4bd6-8a72-50ca08dbc9e2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:02:04.115144 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.115129    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-token" (OuterVolumeSpecName: "istio-token") pod "81c18115-32a9-4bd6-8a72-50ca08dbc9e2" (UID: "81c18115-32a9-4bd6-8a72-50ca08dbc9e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:02:04.115229 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.115177    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "81c18115-32a9-4bd6-8a72-50ca08dbc9e2" (UID: "81c18115-32a9-4bd6-8a72-50ca08dbc9e2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:02:04.115229 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.115205    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-kube-api-access-kn99t" (OuterVolumeSpecName: "kube-api-access-kn99t") pod "81c18115-32a9-4bd6-8a72-50ca08dbc9e2" (UID: "81c18115-32a9-4bd6-8a72-50ca08dbc9e2"). InnerVolumeSpecName "kube-api-access-kn99t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:02:04.213800 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.213751    2556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-csr-ca-configmap\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:02:04.213800 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.213795    2556 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-local-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:02:04.213800 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.213806    2556 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-kubeconfig\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:02:04.213800 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.213814    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn99t\" (UniqueName: \"kubernetes.io/projected/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-kube-api-access-kn99t\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:02:04.213800 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.213823    2556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-csr-dns-cert\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:02:04.214103 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.213832    2556 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-cacerts\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:02:04.214103 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.213840    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81c18115-32a9-4bd6-8a72-50ca08dbc9e2-istio-token\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:02:04.870370 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.870338    2556 generic.go:356] "Generic (PLEG): container finished" podID="81c18115-32a9-4bd6-8a72-50ca08dbc9e2" containerID="2515a111fc2f28e51479984568c7a55851710d2f658268ebe8e8c39df20c38d9" exitCode=0
Sep 09 16:02:04.870829 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.870385    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"
Sep 09 16:02:04.870829 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.870432    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb" event={"ID":"81c18115-32a9-4bd6-8a72-50ca08dbc9e2","Type":"ContainerDied","Data":"2515a111fc2f28e51479984568c7a55851710d2f658268ebe8e8c39df20c38d9"}
Sep 09 16:02:04.870829 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.870501    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb" event={"ID":"81c18115-32a9-4bd6-8a72-50ca08dbc9e2","Type":"ContainerDied","Data":"adce4ed505aee16e82c34af4ad055c2d9163fbc588b9ad5227855c21eef509df"}
Sep 09 16:02:04.870829 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.870525    2556 scope.go:117] "RemoveContainer" containerID="2515a111fc2f28e51479984568c7a55851710d2f658268ebe8e8c39df20c38d9"
Sep 09 16:02:04.878929 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.878907    2556 scope.go:117] "RemoveContainer" containerID="2515a111fc2f28e51479984568c7a55851710d2f658268ebe8e8c39df20c38d9"
Sep 09 16:02:04.879223 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:02:04.879204    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2515a111fc2f28e51479984568c7a55851710d2f658268ebe8e8c39df20c38d9\": container with ID starting with 2515a111fc2f28e51479984568c7a55851710d2f658268ebe8e8c39df20c38d9 not found: ID does not exist" containerID="2515a111fc2f28e51479984568c7a55851710d2f658268ebe8e8c39df20c38d9"
Sep 09 16:02:04.879274 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.879232    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2515a111fc2f28e51479984568c7a55851710d2f658268ebe8e8c39df20c38d9"} err="failed to get container status \"2515a111fc2f28e51479984568c7a55851710d2f658268ebe8e8c39df20c38d9\": rpc error: code = NotFound desc = could not find container \"2515a111fc2f28e51479984568c7a55851710d2f658268ebe8e8c39df20c38d9\": container with ID starting with 2515a111fc2f28e51479984568c7a55851710d2f658268ebe8e8c39df20c38d9 not found: ID does not exist"
Sep 09 16:02:04.896674 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.896640    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"]
Sep 09 16:02:04.903341 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:04.903313    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q7grb"]
Sep 09 16:02:05.589750 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:05.589717    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:02:05.727848 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:05.727810    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:02:05.762541 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:05.762502    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81c18115-32a9-4bd6-8a72-50ca08dbc9e2" path="/var/lib/kubelet/pods/81c18115-32a9-4bd6-8a72-50ca08dbc9e2/volumes"
Sep 09 16:02:14.872604 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:14.872570    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-gw5mr"
Sep 09 16:02:52.458043 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:52.458008    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:02:52.568525 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:02:52.568491    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:27.447385 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:27.447348    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:30.267453 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:30.267408    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:32.050745 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:32.050708    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:32.463886 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:32.463845    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:32.682044 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:32.682008    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:32.853293 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:32.853201    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:34.098414 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:34.098372    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:35.099455 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:35.099416    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:35.221781 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:35.221745    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:36.041165 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:36.041130    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:36.537401 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:36.537362    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:36.748935 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:36.748900    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:36.886203 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:36.886103    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:39.853929 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:39.853896    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:39.951134 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:39.951095    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:42.859582 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.859542    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:42.926786 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.926753    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr"]
Sep 09 16:03:42.927021 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.927009    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81c18115-32a9-4bd6-8a72-50ca08dbc9e2" containerName="discovery"
Sep 09 16:03:42.927069 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.927022    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="81c18115-32a9-4bd6-8a72-50ca08dbc9e2" containerName="discovery"
Sep 09 16:03:42.927107 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.927073    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="81c18115-32a9-4bd6-8a72-50ca08dbc9e2" containerName="discovery"
Sep 09 16:03:42.929880 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.929860    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr"
Sep 09 16:03:42.933870 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.933846    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 16:03:42.933985 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.933908    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 16:03:42.934047 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.933993    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2mfjz\""
Sep 09 16:03:42.934706 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.934689    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 16:03:42.934789 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.934719    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 16:03:42.942231 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.942205    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr"]
Sep 09 16:03:42.949257 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.949231    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:42.970243 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.970209    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc269cb7-8c85-49e5-96cd-aac33a0f7cf8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9vfgr\" (UID: \"bc269cb7-8c85-49e5-96cd-aac33a0f7cf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr"
Sep 09 16:03:42.970243 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.970245    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc269cb7-8c85-49e5-96cd-aac33a0f7cf8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9vfgr\" (UID: \"bc269cb7-8c85-49e5-96cd-aac33a0f7cf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr"
Sep 09 16:03:42.970455 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:42.970267    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9ds7\" (UniqueName: \"kubernetes.io/projected/bc269cb7-8c85-49e5-96cd-aac33a0f7cf8-kube-api-access-q9ds7\") pod \"kuadrant-console-plugin-fbdd4f445-9vfgr\" (UID: \"bc269cb7-8c85-49e5-96cd-aac33a0f7cf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr"
Sep 09 16:03:43.071066 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:43.071023    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc269cb7-8c85-49e5-96cd-aac33a0f7cf8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9vfgr\" (UID: \"bc269cb7-8c85-49e5-96cd-aac33a0f7cf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr"
Sep 09 16:03:43.071241 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:43.071072    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc269cb7-8c85-49e5-96cd-aac33a0f7cf8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9vfgr\" (UID: \"bc269cb7-8c85-49e5-96cd-aac33a0f7cf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr"
Sep 09 16:03:43.071241 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:43.071106    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9ds7\" (UniqueName: \"kubernetes.io/projected/bc269cb7-8c85-49e5-96cd-aac33a0f7cf8-kube-api-access-q9ds7\") pod \"kuadrant-console-plugin-fbdd4f445-9vfgr\" (UID: \"bc269cb7-8c85-49e5-96cd-aac33a0f7cf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr"
Sep 09 16:03:43.071311 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:03:43.071242    2556 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 09 16:03:43.071349 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:03:43.071338    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc269cb7-8c85-49e5-96cd-aac33a0f7cf8-plugin-serving-cert podName:bc269cb7-8c85-49e5-96cd-aac33a0f7cf8 nodeName:}" failed. No retries permitted until 2026-09-09 16:03:43.571316053 +0000 UTC m=+728.437827265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/bc269cb7-8c85-49e5-96cd-aac33a0f7cf8-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-9vfgr" (UID: "bc269cb7-8c85-49e5-96cd-aac33a0f7cf8") : secret "plugin-serving-cert" not found
Sep 09 16:03:43.071828 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:43.071809    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc269cb7-8c85-49e5-96cd-aac33a0f7cf8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9vfgr\" (UID: \"bc269cb7-8c85-49e5-96cd-aac33a0f7cf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr"
Sep 09 16:03:43.080174 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:43.080142    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9ds7\" (UniqueName: \"kubernetes.io/projected/bc269cb7-8c85-49e5-96cd-aac33a0f7cf8-kube-api-access-q9ds7\") pod \"kuadrant-console-plugin-fbdd4f445-9vfgr\" (UID: \"bc269cb7-8c85-49e5-96cd-aac33a0f7cf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr"
Sep 09 16:03:43.300968 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:43.300928    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:43.575105 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:43.575016    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc269cb7-8c85-49e5-96cd-aac33a0f7cf8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9vfgr\" (UID: \"bc269cb7-8c85-49e5-96cd-aac33a0f7cf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr"
Sep 09 16:03:43.577438 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:43.577415    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc269cb7-8c85-49e5-96cd-aac33a0f7cf8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9vfgr\" (UID: \"bc269cb7-8c85-49e5-96cd-aac33a0f7cf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr"
Sep 09 16:03:43.838637 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:43.838554    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr"
Sep 09 16:03:44.110070 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:44.110040    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr"]
Sep 09 16:03:44.112389 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:03:44.112353    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc269cb7_8c85_49e5_96cd_aac33a0f7cf8.slice/crio-1c32338c31a03c10e6f558f71f4e9b3c1f6a20d8f13f34dc4da89cf385c3273e WatchSource:0}: Error finding container 1c32338c31a03c10e6f558f71f4e9b3c1f6a20d8f13f34dc4da89cf385c3273e: Status 404 returned error can't find the container with id 1c32338c31a03c10e6f558f71f4e9b3c1f6a20d8f13f34dc4da89cf385c3273e
Sep 09 16:03:44.113745 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:44.113727    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:03:44.237474 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:44.237419    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr" event={"ID":"bc269cb7-8c85-49e5-96cd-aac33a0f7cf8","Type":"ContainerStarted","Data":"1c32338c31a03c10e6f558f71f4e9b3c1f6a20d8f13f34dc4da89cf385c3273e"}
Sep 09 16:03:47.603116 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:47.603084    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:47.717668 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:47.717630    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:47.777369 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:47.777334    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:50.513551 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:50.513513    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:57.736033 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:57.735992    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:57.833908 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:57.833802    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:58.293903 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:58.293863    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr" event={"ID":"bc269cb7-8c85-49e5-96cd-aac33a0f7cf8","Type":"ContainerStarted","Data":"6dfdde6233eb52452791bdfbc840466e07357d6256cd53cc620c9cdda111abf2"}
Sep 09 16:03:58.308594 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:58.308540    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vfgr" podStartSLOduration=2.879361152 podStartE2EDuration="16.308522914s" totalImagesPullingTime="13.429161762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:03:58.307047961 +0000 UTC m=+743.173559180" watchObservedRunningTime="2026-09-09 16:03:58.308522914 +0000 UTC m=+743.175034133"
Sep 09 16:03:59.049714 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:59.049673    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:03:59.133859 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:03:59.133819    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:02.033017 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:02.032985    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:02.134851 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:02.134816    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:02.477174 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:02.477135    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:03.906161 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:03.906124    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:10.153679 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:10.153643    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:10.212292 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:10.212253    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:15.605277 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:15.605226    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:15.705239 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:15.705203    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:22.770018 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:22.769980    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:22.870899 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:22.870863    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:36.124205 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:36.124167    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:36.219491 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:36.219435    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:36.279326 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:36.279293    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:36.895334 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:36.895301    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-2xkrm"]
Sep 09 16:04:36.899945 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:36.899923    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-2xkrm"
Sep 09 16:04:36.902169 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:36.902147    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bsv9f\""
Sep 09 16:04:36.905956 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:36.905933    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-2xkrm"]
Sep 09 16:04:36.931879 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:36.931829    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:36.986000 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:36.985964    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dffm\" (UniqueName: \"kubernetes.io/projected/3661ed63-9ac7-48a4-814b-9df37c5a12e7-kube-api-access-2dffm\") pod \"authorino-7bcfd8f9cd-2xkrm\" (UID: \"3661ed63-9ac7-48a4-814b-9df37c5a12e7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-2xkrm"
Sep 09 16:04:37.006823 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:37.006778    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:37.007058 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:37.007041    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-sws94"]
Sep 09 16:04:37.008996 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:37.008978    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sws94"
Sep 09 16:04:37.013617 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:37.013591    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sws94"]
Sep 09 16:04:37.086867 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:37.086829    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lkvd\" (UniqueName: \"kubernetes.io/projected/e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce-kube-api-access-8lkvd\") pod \"authorino-59c974bd48-sws94\" (UID: \"e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce\") " pod="kuadrant-system/authorino-59c974bd48-sws94"
Sep 09 16:04:37.087041 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:37.086893    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dffm\" (UniqueName: \"kubernetes.io/projected/3661ed63-9ac7-48a4-814b-9df37c5a12e7-kube-api-access-2dffm\") pod \"authorino-7bcfd8f9cd-2xkrm\" (UID: \"3661ed63-9ac7-48a4-814b-9df37c5a12e7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-2xkrm"
Sep 09 16:04:37.096646 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:37.096623    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dffm\" (UniqueName: \"kubernetes.io/projected/3661ed63-9ac7-48a4-814b-9df37c5a12e7-kube-api-access-2dffm\") pod \"authorino-7bcfd8f9cd-2xkrm\" (UID: \"3661ed63-9ac7-48a4-814b-9df37c5a12e7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-2xkrm"
Sep 09 16:04:37.187586 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:37.187551    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lkvd\" (UniqueName: \"kubernetes.io/projected/e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce-kube-api-access-8lkvd\") pod \"authorino-59c974bd48-sws94\" (UID: \"e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce\") " pod="kuadrant-system/authorino-59c974bd48-sws94"
Sep 09 16:04:37.198352 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:37.198320    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lkvd\" (UniqueName: \"kubernetes.io/projected/e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce-kube-api-access-8lkvd\") pod \"authorino-59c974bd48-sws94\" (UID: \"e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce\") " pod="kuadrant-system/authorino-59c974bd48-sws94"
Sep 09 16:04:37.209969 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:37.209938    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-2xkrm"
Sep 09 16:04:37.318086 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:37.318039    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sws94"
Sep 09 16:04:37.475947 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:37.475758    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-2xkrm"]
Sep 09 16:04:37.478366 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:04:37.478337    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3661ed63_9ac7_48a4_814b_9df37c5a12e7.slice/crio-ae9bf75399efd214717f38ad03320953118032f6db307bb51eedc296c6f9f67c WatchSource:0}: Error finding container ae9bf75399efd214717f38ad03320953118032f6db307bb51eedc296c6f9f67c: Status 404 returned error can't find the container with id ae9bf75399efd214717f38ad03320953118032f6db307bb51eedc296c6f9f67c
Sep 09 16:04:37.582437 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:37.582406    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sws94"]
Sep 09 16:04:37.585458 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:04:37.585423    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8e516c9_11a6_4fb3_a1cf_08b0bae5a5ce.slice/crio-7b191340dd61a5063c3d56cd6d2b4b693a8e04f93d53633c9a869dffa17a1ecd WatchSource:0}: Error finding container 7b191340dd61a5063c3d56cd6d2b4b693a8e04f93d53633c9a869dffa17a1ecd: Status 404 returned error can't find the container with id 7b191340dd61a5063c3d56cd6d2b4b693a8e04f93d53633c9a869dffa17a1ecd
Sep 09 16:04:38.438637 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:38.438601    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sws94" event={"ID":"e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce","Type":"ContainerStarted","Data":"7b191340dd61a5063c3d56cd6d2b4b693a8e04f93d53633c9a869dffa17a1ecd"}
Sep 09 16:04:38.439757 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:38.439732    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-2xkrm" event={"ID":"3661ed63-9ac7-48a4-814b-9df37c5a12e7","Type":"ContainerStarted","Data":"ae9bf75399efd214717f38ad03320953118032f6db307bb51eedc296c6f9f67c"}
Sep 09 16:04:41.455579 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:41.455538    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sws94" event={"ID":"e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce","Type":"ContainerStarted","Data":"60241de32a5f82787510ee885720e751e88fd8392a88a44b6d6b5d6061d892c4"}
Sep 09 16:04:41.457044 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:41.457015    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-2xkrm" event={"ID":"3661ed63-9ac7-48a4-814b-9df37c5a12e7","Type":"ContainerStarted","Data":"b452717e471653920e49f72beaeceec3072d2e84b93f5602b1e2fec873b1d415"}
Sep 09 16:04:41.470804 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:41.470755    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-sws94" podStartSLOduration=2.475341506 podStartE2EDuration="5.470740169s" totalImagesPullingTime="2.995398663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:04:41.469220741 +0000 UTC m=+786.335731960" watchObservedRunningTime="2026-09-09 16:04:41.470740169 +0000 UTC m=+786.337251390"
Sep 09 16:04:41.483284 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:41.483224    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-2xkrm" podStartSLOduration=2.370640555 podStartE2EDuration="5.483206057s" totalImagesPullingTime="3.112565502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:04:41.481734146 +0000 UTC m=+786.348245366" watchObservedRunningTime="2026-09-09 16:04:41.483206057 +0000 UTC m=+786.349717275"
Sep 09 16:04:41.507196 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:41.507158    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-2xkrm"]
Sep 09 16:04:43.466611 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:43.466566    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-2xkrm" podUID="3661ed63-9ac7-48a4-814b-9df37c5a12e7" containerName="authorino" containerID="cri-o://b452717e471653920e49f72beaeceec3072d2e84b93f5602b1e2fec873b1d415" gracePeriod=30
Sep 09 16:04:43.717786 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:43.717722    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-2xkrm"
Sep 09 16:04:43.744246 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:43.744216    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dffm\" (UniqueName: \"kubernetes.io/projected/3661ed63-9ac7-48a4-814b-9df37c5a12e7-kube-api-access-2dffm\") pod \"3661ed63-9ac7-48a4-814b-9df37c5a12e7\" (UID: \"3661ed63-9ac7-48a4-814b-9df37c5a12e7\") "
Sep 09 16:04:43.746484 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:43.746431    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3661ed63-9ac7-48a4-814b-9df37c5a12e7-kube-api-access-2dffm" (OuterVolumeSpecName: "kube-api-access-2dffm") pod "3661ed63-9ac7-48a4-814b-9df37c5a12e7" (UID: "3661ed63-9ac7-48a4-814b-9df37c5a12e7"). InnerVolumeSpecName "kube-api-access-2dffm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:04:43.845420 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:43.845384    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dffm\" (UniqueName: \"kubernetes.io/projected/3661ed63-9ac7-48a4-814b-9df37c5a12e7-kube-api-access-2dffm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:04:44.471518 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:44.471453    2556 generic.go:356] "Generic (PLEG): container finished" podID="3661ed63-9ac7-48a4-814b-9df37c5a12e7" containerID="b452717e471653920e49f72beaeceec3072d2e84b93f5602b1e2fec873b1d415" exitCode=0
Sep 09 16:04:44.471518 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:44.471495    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-2xkrm" event={"ID":"3661ed63-9ac7-48a4-814b-9df37c5a12e7","Type":"ContainerDied","Data":"b452717e471653920e49f72beaeceec3072d2e84b93f5602b1e2fec873b1d415"}
Sep 09 16:04:44.472035 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:44.471541    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-2xkrm" event={"ID":"3661ed63-9ac7-48a4-814b-9df37c5a12e7","Type":"ContainerDied","Data":"ae9bf75399efd214717f38ad03320953118032f6db307bb51eedc296c6f9f67c"}
Sep 09 16:04:44.472035 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:44.471559    2556 scope.go:117] "RemoveContainer" containerID="b452717e471653920e49f72beaeceec3072d2e84b93f5602b1e2fec873b1d415"
Sep 09 16:04:44.472035 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:44.471508    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-2xkrm"
Sep 09 16:04:44.479365 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:44.479343    2556 scope.go:117] "RemoveContainer" containerID="b452717e471653920e49f72beaeceec3072d2e84b93f5602b1e2fec873b1d415"
Sep 09 16:04:44.479662 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:04:44.479624    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b452717e471653920e49f72beaeceec3072d2e84b93f5602b1e2fec873b1d415\": container with ID starting with b452717e471653920e49f72beaeceec3072d2e84b93f5602b1e2fec873b1d415 not found: ID does not exist" containerID="b452717e471653920e49f72beaeceec3072d2e84b93f5602b1e2fec873b1d415"
Sep 09 16:04:44.479711 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:44.479655    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b452717e471653920e49f72beaeceec3072d2e84b93f5602b1e2fec873b1d415"} err="failed to get container status \"b452717e471653920e49f72beaeceec3072d2e84b93f5602b1e2fec873b1d415\": rpc error: code = NotFound desc = could not find container \"b452717e471653920e49f72beaeceec3072d2e84b93f5602b1e2fec873b1d415\": container with ID starting with b452717e471653920e49f72beaeceec3072d2e84b93f5602b1e2fec873b1d415 not found: ID does not exist"
Sep 09 16:04:44.487138 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:44.487109    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-2xkrm"]
Sep 09 16:04:44.490588 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:44.490568    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-2xkrm"]
Sep 09 16:04:45.762385 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:45.762353    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3661ed63-9ac7-48a4-814b-9df37c5a12e7" path="/var/lib/kubelet/pods/3661ed63-9ac7-48a4-814b-9df37c5a12e7/volumes"
Sep 09 16:04:51.765114 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:51.765072    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:51.866237 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:51.866199    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:52.284975 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:04:52.284937    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:05.037397 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.037301    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:05.278183 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.278145    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-xjh54"]
Sep 09 16:05:05.278420 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.278408    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3661ed63-9ac7-48a4-814b-9df37c5a12e7" containerName="authorino"
Sep 09 16:05:05.278458 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.278422    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3661ed63-9ac7-48a4-814b-9df37c5a12e7" containerName="authorino"
Sep 09 16:05:05.278547 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.278479    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3661ed63-9ac7-48a4-814b-9df37c5a12e7" containerName="authorino"
Sep 09 16:05:05.281196 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.281177    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xjh54"
Sep 09 16:05:05.288807 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.288749    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-xjh54"]
Sep 09 16:05:05.344226 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.344185    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-xjh54"]
Sep 09 16:05:05.344458 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:05:05.344430    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4l8z4], unattached volumes=[], failed to process volumes=[kube-api-access-4l8z4]: context canceled" pod="kuadrant-system/authorino-557f876f69-xjh54" podUID="8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799"
Sep 09 16:05:05.370299 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.370263    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9lwff"]
Sep 09 16:05:05.373263 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.373244    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9lwff"
Sep 09 16:05:05.375710 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.375690    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 16:05:05.380681 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.380654    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9lwff"]
Sep 09 16:05:05.417044 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.416976    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l8z4\" (UniqueName: \"kubernetes.io/projected/8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799-kube-api-access-4l8z4\") pod \"authorino-557f876f69-xjh54\" (UID: \"8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799\") " pod="kuadrant-system/authorino-557f876f69-xjh54"
Sep 09 16:05:05.517671 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.517631    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l8z4\" (UniqueName: \"kubernetes.io/projected/8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799-kube-api-access-4l8z4\") pod \"authorino-557f876f69-xjh54\" (UID: \"8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799\") " pod="kuadrant-system/authorino-557f876f69-xjh54"
Sep 09 16:05:05.517846 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.517686    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh6b9\" (UniqueName: \"kubernetes.io/projected/a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0-kube-api-access-zh6b9\") pod \"authorino-6d4dfc649f-9lwff\" (UID: \"a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0\") " pod="kuadrant-system/authorino-6d4dfc649f-9lwff"
Sep 09 16:05:05.517846 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.517744    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0-tls-cert\") pod \"authorino-6d4dfc649f-9lwff\" (UID: \"a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0\") " pod="kuadrant-system/authorino-6d4dfc649f-9lwff"
Sep 09 16:05:05.525474 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.525435    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l8z4\" (UniqueName: \"kubernetes.io/projected/8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799-kube-api-access-4l8z4\") pod \"authorino-557f876f69-xjh54\" (UID: \"8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799\") " pod="kuadrant-system/authorino-557f876f69-xjh54"
Sep 09 16:05:05.542676 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.542589    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9lwff"]
Sep 09 16:05:05.542856 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:05:05.542834    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zh6b9 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-9lwff" podUID="a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0"
Sep 09 16:05:05.556784 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.556751    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9lwff"
Sep 09 16:05:05.556951 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.556756    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xjh54"
Sep 09 16:05:05.564014 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.563990    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9lwff"
Sep 09 16:05:05.566929 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.566899    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xjh54"
Sep 09 16:05:05.568921 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.568899    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-647d78969f-f5mzk"]
Sep 09 16:05:05.571820 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.571801    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-647d78969f-f5mzk"
Sep 09 16:05:05.582007 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.581981    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-647d78969f-f5mzk"]
Sep 09 16:05:05.618256 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.618217    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l8z4\" (UniqueName: \"kubernetes.io/projected/8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799-kube-api-access-4l8z4\") pod \"8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799\" (UID: \"8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799\") "
Sep 09 16:05:05.618404 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.618318    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh6b9\" (UniqueName: \"kubernetes.io/projected/a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0-kube-api-access-zh6b9\") pod \"authorino-6d4dfc649f-9lwff\" (UID: \"a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0\") " pod="kuadrant-system/authorino-6d4dfc649f-9lwff"
Sep 09 16:05:05.618404 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.618391    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0-tls-cert\") pod \"authorino-6d4dfc649f-9lwff\" (UID: \"a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0\") " pod="kuadrant-system/authorino-6d4dfc649f-9lwff"
Sep 09 16:05:05.620422 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.620388    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799-kube-api-access-4l8z4" (OuterVolumeSpecName: "kube-api-access-4l8z4") pod "8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799" (UID: "8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799"). InnerVolumeSpecName "kube-api-access-4l8z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:05:05.620902 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.620879    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0-tls-cert\") pod \"authorino-6d4dfc649f-9lwff\" (UID: \"a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0\") " pod="kuadrant-system/authorino-6d4dfc649f-9lwff"
Sep 09 16:05:05.625107 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.625075    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh6b9\" (UniqueName: \"kubernetes.io/projected/a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0-kube-api-access-zh6b9\") pod \"authorino-6d4dfc649f-9lwff\" (UID: \"a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0\") " pod="kuadrant-system/authorino-6d4dfc649f-9lwff"
Sep 09 16:05:05.719342 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.719297    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh6b9\" (UniqueName: \"kubernetes.io/projected/a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0-kube-api-access-zh6b9\") pod \"a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0\" (UID: \"a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0\") "
Sep 09 16:05:05.719531 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.719412    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0-tls-cert\") pod \"a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0\" (UID: \"a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0\") "
Sep 09 16:05:05.719587 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.719552    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77e64e31-d52e-4e26-985a-1ec780faebfb-tls-cert\") pod \"authorino-647d78969f-f5mzk\" (UID: \"77e64e31-d52e-4e26-985a-1ec780faebfb\") " pod="kuadrant-system/authorino-647d78969f-f5mzk"
Sep 09 16:05:05.719657 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.719639    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-699b2\" (UniqueName: \"kubernetes.io/projected/77e64e31-d52e-4e26-985a-1ec780faebfb-kube-api-access-699b2\") pod \"authorino-647d78969f-f5mzk\" (UID: \"77e64e31-d52e-4e26-985a-1ec780faebfb\") " pod="kuadrant-system/authorino-647d78969f-f5mzk"
Sep 09 16:05:05.719763 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.719737    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l8z4\" (UniqueName: \"kubernetes.io/projected/8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799-kube-api-access-4l8z4\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:05:05.721598 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.721569    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0-kube-api-access-zh6b9" (OuterVolumeSpecName: "kube-api-access-zh6b9") pod "a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0" (UID: "a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0"). InnerVolumeSpecName "kube-api-access-zh6b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:05:05.721693 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.721577    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0" (UID: "a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:05:05.821040 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.820927    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-699b2\" (UniqueName: \"kubernetes.io/projected/77e64e31-d52e-4e26-985a-1ec780faebfb-kube-api-access-699b2\") pod \"authorino-647d78969f-f5mzk\" (UID: \"77e64e31-d52e-4e26-985a-1ec780faebfb\") " pod="kuadrant-system/authorino-647d78969f-f5mzk"
Sep 09 16:05:05.821040 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.821017    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77e64e31-d52e-4e26-985a-1ec780faebfb-tls-cert\") pod \"authorino-647d78969f-f5mzk\" (UID: \"77e64e31-d52e-4e26-985a-1ec780faebfb\") " pod="kuadrant-system/authorino-647d78969f-f5mzk"
Sep 09 16:05:05.821239 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.821057    2556 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0-tls-cert\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:05:05.821239 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.821073    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh6b9\" (UniqueName: \"kubernetes.io/projected/a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0-kube-api-access-zh6b9\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:05:05.823597 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.823572    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77e64e31-d52e-4e26-985a-1ec780faebfb-tls-cert\") pod \"authorino-647d78969f-f5mzk\" (UID: \"77e64e31-d52e-4e26-985a-1ec780faebfb\") " pod="kuadrant-system/authorino-647d78969f-f5mzk"
Sep 09 16:05:05.829821 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.829793    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-699b2\" (UniqueName: \"kubernetes.io/projected/77e64e31-d52e-4e26-985a-1ec780faebfb-kube-api-access-699b2\") pod \"authorino-647d78969f-f5mzk\" (UID: \"77e64e31-d52e-4e26-985a-1ec780faebfb\") " pod="kuadrant-system/authorino-647d78969f-f5mzk"
Sep 09 16:05:05.880761 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:05.880725    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-647d78969f-f5mzk"
Sep 09 16:05:06.149674 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:06.149592    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-647d78969f-f5mzk"]
Sep 09 16:05:06.152443 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:05:06.152412    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77e64e31_d52e_4e26_985a_1ec780faebfb.slice/crio-307f966c29925f9fe59c4780a9f08f34ab7a619b064609a0126ccaef952bb1a6 WatchSource:0}: Error finding container 307f966c29925f9fe59c4780a9f08f34ab7a619b064609a0126ccaef952bb1a6: Status 404 returned error can't find the container with id 307f966c29925f9fe59c4780a9f08f34ab7a619b064609a0126ccaef952bb1a6
Sep 09 16:05:06.562102 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:06.562066    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xjh54"
Sep 09 16:05:06.562102 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:06.562100    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9lwff"
Sep 09 16:05:06.562305 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:06.562075    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-647d78969f-f5mzk" event={"ID":"77e64e31-d52e-4e26-985a-1ec780faebfb","Type":"ContainerStarted","Data":"307f966c29925f9fe59c4780a9f08f34ab7a619b064609a0126ccaef952bb1a6"}
Sep 09 16:05:06.588316 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:06.588281    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9lwff"]
Sep 09 16:05:06.592010 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:06.591979    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9lwff"]
Sep 09 16:05:06.611142 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:06.611104    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-xjh54"]
Sep 09 16:05:06.614724 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:06.614697    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-xjh54"]
Sep 09 16:05:07.762913 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:07.762879    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799" path="/var/lib/kubelet/pods/8ed4ebad-8a04-4fd8-a7b8-f366d8eaf799/volumes"
Sep 09 16:05:07.763374 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:07.763119    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0" path="/var/lib/kubelet/pods/a6b5cdbb-891e-428b-b7b5-bb8dbd3af9e0/volumes"
Sep 09 16:05:08.571169 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:08.571134    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-647d78969f-f5mzk" event={"ID":"77e64e31-d52e-4e26-985a-1ec780faebfb","Type":"ContainerStarted","Data":"4614950a9c021c4d26e1300e1a453177abddd983733f2a2e30a798af594c296d"}
Sep 09 16:05:08.587020 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:08.586974    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-647d78969f-f5mzk" podStartSLOduration=1.858739251 podStartE2EDuration="3.586957503s" totalImagesPullingTime="1.728218252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:05:08.585382202 +0000 UTC m=+813.451893421" watchObservedRunningTime="2026-09-09 16:05:08.586957503 +0000 UTC m=+813.453468730"
Sep 09 16:05:08.608388 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:08.608344    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sws94"]
Sep 09 16:05:08.608589 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:08.608566    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-sws94" podUID="e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce" containerName="authorino" containerID="cri-o://60241de32a5f82787510ee885720e751e88fd8392a88a44b6d6b5d6061d892c4" gracePeriod=30
Sep 09 16:05:08.672209 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:08.672174    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:08.840482 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:08.840442    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sws94"
Sep 09 16:05:08.948102 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:08.948056    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lkvd\" (UniqueName: \"kubernetes.io/projected/e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce-kube-api-access-8lkvd\") pod \"e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce\" (UID: \"e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce\") "
Sep 09 16:05:08.950080 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:08.950052    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce-kube-api-access-8lkvd" (OuterVolumeSpecName: "kube-api-access-8lkvd") pod "e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce" (UID: "e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce"). InnerVolumeSpecName "kube-api-access-8lkvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:05:09.049423 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:09.049380    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lkvd\" (UniqueName: \"kubernetes.io/projected/e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce-kube-api-access-8lkvd\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:05:09.577178 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:09.577142    2556 generic.go:356] "Generic (PLEG): container finished" podID="e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce" containerID="60241de32a5f82787510ee885720e751e88fd8392a88a44b6d6b5d6061d892c4" exitCode=0
Sep 09 16:05:09.577178 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:09.577182    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sws94"
Sep 09 16:05:09.577383 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:09.577230    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sws94" event={"ID":"e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce","Type":"ContainerDied","Data":"60241de32a5f82787510ee885720e751e88fd8392a88a44b6d6b5d6061d892c4"}
Sep 09 16:05:09.577383 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:09.577272    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sws94" event={"ID":"e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce","Type":"ContainerDied","Data":"7b191340dd61a5063c3d56cd6d2b4b693a8e04f93d53633c9a869dffa17a1ecd"}
Sep 09 16:05:09.577383 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:09.577293    2556 scope.go:117] "RemoveContainer" containerID="60241de32a5f82787510ee885720e751e88fd8392a88a44b6d6b5d6061d892c4"
Sep 09 16:05:09.585188 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:09.585171    2556 scope.go:117] "RemoveContainer" containerID="60241de32a5f82787510ee885720e751e88fd8392a88a44b6d6b5d6061d892c4"
Sep 09 16:05:09.585436 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:05:09.585416    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60241de32a5f82787510ee885720e751e88fd8392a88a44b6d6b5d6061d892c4\": container with ID starting with 60241de32a5f82787510ee885720e751e88fd8392a88a44b6d6b5d6061d892c4 not found: ID does not exist" containerID="60241de32a5f82787510ee885720e751e88fd8392a88a44b6d6b5d6061d892c4"
Sep 09 16:05:09.585554 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:09.585448    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60241de32a5f82787510ee885720e751e88fd8392a88a44b6d6b5d6061d892c4"} err="failed to get container status \"60241de32a5f82787510ee885720e751e88fd8392a88a44b6d6b5d6061d892c4\": rpc error: code = NotFound desc = could not find container \"60241de32a5f82787510ee885720e751e88fd8392a88a44b6d6b5d6061d892c4\": container with ID starting with 60241de32a5f82787510ee885720e751e88fd8392a88a44b6d6b5d6061d892c4 not found: ID does not exist"
Sep 09 16:05:09.595249 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:09.595223    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sws94"]
Sep 09 16:05:09.599014 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:09.598992    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sws94"]
Sep 09 16:05:09.762846 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:09.762813    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce" path="/var/lib/kubelet/pods/e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce/volumes"
Sep 09 16:05:13.673584 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:13.673553    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:13.755430 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:13.755390    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:13.812635 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:13.812598    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:14.057520 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:14.057412    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:14.168643 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:14.168607    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:17.857519 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:17.857484    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:17.970162 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:17.970131    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:49.598522 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:49.598479    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:49.696640 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:49.696601    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:51.159827 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.159784    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:51.239570 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.239535    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:51.397909 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.397873    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:51.498366 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.498331    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:51.654609 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.654573    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:51.759305 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.759206    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:51.771970 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.771938    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c"]
Sep 09 16:05:51.772400 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.772380    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce" containerName="authorino"
Sep 09 16:05:51.772400 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.772401    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce" containerName="authorino"
Sep 09 16:05:51.772582 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.772502    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e516c9-11a6-4fb3-a1cf-08b0bae5a5ce" containerName="authorino"
Sep 09 16:05:51.776778 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.776753    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c"
Sep 09 16:05:51.779171 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.779146    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 16:05:51.779171 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.779161    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-66cvm\""
Sep 09 16:05:51.789439 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.789410    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c"]
Sep 09 16:05:51.872390 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.872350    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h5rt\" (UniqueName: \"kubernetes.io/projected/d6653c3a-2dbc-41af-8b21-9ca857ba26f4-kube-api-access-5h5rt\") pod \"payload-pre-processing-59fbc47bf8-zw24c\" (UID: \"d6653c3a-2dbc-41af-8b21-9ca857ba26f4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c"
Sep 09 16:05:51.872599 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.872413    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6653c3a-2dbc-41af-8b21-9ca857ba26f4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-zw24c\" (UID: \"d6653c3a-2dbc-41af-8b21-9ca857ba26f4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c"
Sep 09 16:05:51.973789 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.973750    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6653c3a-2dbc-41af-8b21-9ca857ba26f4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-zw24c\" (UID: \"d6653c3a-2dbc-41af-8b21-9ca857ba26f4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c"
Sep 09 16:05:51.974034 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.973809    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h5rt\" (UniqueName: \"kubernetes.io/projected/d6653c3a-2dbc-41af-8b21-9ca857ba26f4-kube-api-access-5h5rt\") pod \"payload-pre-processing-59fbc47bf8-zw24c\" (UID: \"d6653c3a-2dbc-41af-8b21-9ca857ba26f4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c"
Sep 09 16:05:51.979163 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.975155    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6653c3a-2dbc-41af-8b21-9ca857ba26f4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-zw24c\" (UID: \"d6653c3a-2dbc-41af-8b21-9ca857ba26f4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c"
Sep 09 16:05:51.986526 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:51.986500    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h5rt\" (UniqueName: \"kubernetes.io/projected/d6653c3a-2dbc-41af-8b21-9ca857ba26f4-kube-api-access-5h5rt\") pod \"payload-pre-processing-59fbc47bf8-zw24c\" (UID: \"d6653c3a-2dbc-41af-8b21-9ca857ba26f4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c"
Sep 09 16:05:52.087557 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:52.087457    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c"
Sep 09 16:05:52.356989 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:52.356961    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c"]
Sep 09 16:05:52.359680 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:05:52.359653    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6653c3a_2dbc_41af_8b21_9ca857ba26f4.slice/crio-8cf6fadb5faa509b279b764cec601eb751b19d1ee4a03fc244841aa83aa867ee WatchSource:0}: Error finding container 8cf6fadb5faa509b279b764cec601eb751b19d1ee4a03fc244841aa83aa867ee: Status 404 returned error can't find the container with id 8cf6fadb5faa509b279b764cec601eb751b19d1ee4a03fc244841aa83aa867ee
Sep 09 16:05:52.751930 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:52.751892    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c" event={"ID":"d6653c3a-2dbc-41af-8b21-9ca857ba26f4","Type":"ContainerStarted","Data":"8cf6fadb5faa509b279b764cec601eb751b19d1ee4a03fc244841aa83aa867ee"}
Sep 09 16:05:55.766295 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:55.766260    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c" event={"ID":"d6653c3a-2dbc-41af-8b21-9ca857ba26f4","Type":"ContainerStarted","Data":"8f605c821e4bdc208b71d53758e04b455cf15f77d408d3ce4f928b5fd91bd978"}
Sep 09 16:05:55.766748 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:55.766324    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c"
Sep 09 16:05:55.786429 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:05:55.786377    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c" podStartSLOduration=2.122627827 podStartE2EDuration="4.786362624s" totalImagesPullingTime="2.663734797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:05:55.78489509 +0000 UTC m=+860.651406309" watchObservedRunningTime="2026-09-09 16:05:55.786362624 +0000 UTC m=+860.652873843"
Sep 09 16:06:01.055456 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:01.055417    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:06:01.174800 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:01.174762    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:06:06.772263 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:06.772226    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zw24c"
Sep 09 16:06:08.855331 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:08.855288    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:06:16.036175 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.036135    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"]
Sep 09 16:06:16.039472 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.039440    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.041746 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.041725    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 16:06:16.042344 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.042325    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 09 16:06:16.042443 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.042369    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kzfjj\""
Sep 09 16:06:16.042443 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.042385    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 16:06:16.050088 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.050063    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"]
Sep 09 16:06:16.077046 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.077020    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:06:16.160562 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.160526    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fba577e-4974-4fc1-b73b-65ff011560b5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.160722 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.160580    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fba577e-4974-4fc1-b73b-65ff011560b5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.160722 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.160603    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fba577e-4974-4fc1-b73b-65ff011560b5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.160722 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.160642    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fba577e-4974-4fc1-b73b-65ff011560b5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.160722 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.160665    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkb84\" (UniqueName: \"kubernetes.io/projected/8fba577e-4974-4fc1-b73b-65ff011560b5-kube-api-access-hkb84\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.160722 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.160693    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fba577e-4974-4fc1-b73b-65ff011560b5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.261360 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.261321    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fba577e-4974-4fc1-b73b-65ff011560b5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.261360 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.261362    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fba577e-4974-4fc1-b73b-65ff011560b5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.261632 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.261384    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fba577e-4974-4fc1-b73b-65ff011560b5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.261632 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.261406    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkb84\" (UniqueName: \"kubernetes.io/projected/8fba577e-4974-4fc1-b73b-65ff011560b5-kube-api-access-hkb84\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.261632 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.261434    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fba577e-4974-4fc1-b73b-65ff011560b5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.261632 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.261495    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fba577e-4974-4fc1-b73b-65ff011560b5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.261932 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.261896    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fba577e-4974-4fc1-b73b-65ff011560b5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.261932 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.261925    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fba577e-4974-4fc1-b73b-65ff011560b5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.262087 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.261903    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fba577e-4974-4fc1-b73b-65ff011560b5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.263735 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.263706    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fba577e-4974-4fc1-b73b-65ff011560b5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.263857 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.263838    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fba577e-4974-4fc1-b73b-65ff011560b5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.268606 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.268588    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkb84\" (UniqueName: \"kubernetes.io/projected/8fba577e-4974-4fc1-b73b-65ff011560b5-kube-api-access-hkb84\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth\" (UID: \"8fba577e-4974-4fc1-b73b-65ff011560b5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.350434 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.350338    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:16.624330 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.624242    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"]
Sep 09 16:06:16.629604 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:06:16.629573    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fba577e_4974_4fc1_b73b_65ff011560b5.slice/crio-0e6ced3326be3004bf2401ff8c5d8a08ac188ed322fb259fcd96ecc1e180bba1 WatchSource:0}: Error finding container 0e6ced3326be3004bf2401ff8c5d8a08ac188ed322fb259fcd96ecc1e180bba1: Status 404 returned error can't find the container with id 0e6ced3326be3004bf2401ff8c5d8a08ac188ed322fb259fcd96ecc1e180bba1
Sep 09 16:06:16.849403 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:16.849358    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth" event={"ID":"8fba577e-4974-4fc1-b73b-65ff011560b5","Type":"ContainerStarted","Data":"0e6ced3326be3004bf2401ff8c5d8a08ac188ed322fb259fcd96ecc1e180bba1"}
Sep 09 16:06:18.858900 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:18.858859    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth" event={"ID":"8fba577e-4974-4fc1-b73b-65ff011560b5","Type":"ContainerStarted","Data":"05dda9b8a40fd64d4a032744f0cbd3d5ba6c2170b9b672584e60283de81df570"}
Sep 09 16:06:18.875105 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:18.875055    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth" podStartSLOduration=1.6969543969999998 podStartE2EDuration="2.875038814s" totalImagesPullingTime="1.178084417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:06:18.873824675 +0000 UTC m=+883.740335894" watchObservedRunningTime="2026-09-09 16:06:18.875038814 +0000 UTC m=+883.741550034"
Sep 09 16:06:26.350860 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.350822    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:26.351270 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.350886    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:26.363345 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.363321    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:26.634175 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.634101    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"]
Sep 09 16:06:26.637546 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.637525    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.639633 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.639616    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 09 16:06:26.647566 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.647539    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"]
Sep 09 16:06:26.741564 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.741531    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:06:26.743559 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.743538    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3cd58b7-297c-4b14-b353-e029001f2797-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.743626 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.743575    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3cd58b7-297c-4b14-b353-e029001f2797-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.743626 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.743593    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3cd58b7-297c-4b14-b353-e029001f2797-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.743698 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.743671    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgttk\" (UniqueName: \"kubernetes.io/projected/c3cd58b7-297c-4b14-b353-e029001f2797-kube-api-access-vgttk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.743732 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.743716    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3cd58b7-297c-4b14-b353-e029001f2797-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.743766 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.743743    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3cd58b7-297c-4b14-b353-e029001f2797-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.844381 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.844329    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3cd58b7-297c-4b14-b353-e029001f2797-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.844381 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.844389    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3cd58b7-297c-4b14-b353-e029001f2797-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.844641 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.844405    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3cd58b7-297c-4b14-b353-e029001f2797-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.844641 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.844491    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgttk\" (UniqueName: \"kubernetes.io/projected/c3cd58b7-297c-4b14-b353-e029001f2797-kube-api-access-vgttk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.844641 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.844535    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3cd58b7-297c-4b14-b353-e029001f2797-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.844641 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.844580    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3cd58b7-297c-4b14-b353-e029001f2797-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.844815 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.844789    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3cd58b7-297c-4b14-b353-e029001f2797-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.844882 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.844860    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3cd58b7-297c-4b14-b353-e029001f2797-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.845066 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.845042    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3cd58b7-297c-4b14-b353-e029001f2797-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.846738 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.846717    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3cd58b7-297c-4b14-b353-e029001f2797-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.846957 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.846942    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3cd58b7-297c-4b14-b353-e029001f2797-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.852953 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.852933    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgttk\" (UniqueName: \"kubernetes.io/projected/c3cd58b7-297c-4b14-b353-e029001f2797-kube-api-access-vgttk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pppdx\" (UID: \"c3cd58b7-297c-4b14-b353-e029001f2797\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:26.901697 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.901622    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth"
Sep 09 16:06:26.967033 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:26.966991    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:27.246965 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:27.246918    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"]
Sep 09 16:06:27.251679 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:06:27.251655    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cd58b7_297c_4b14_b353_e029001f2797.slice/crio-b11f26955232698a81a76868bde1be3eaa4e0e6d8fa059ff1ad7901dc562960f WatchSource:0}: Error finding container b11f26955232698a81a76868bde1be3eaa4e0e6d8fa059ff1ad7901dc562960f: Status 404 returned error can't find the container with id b11f26955232698a81a76868bde1be3eaa4e0e6d8fa059ff1ad7901dc562960f
Sep 09 16:06:27.895252 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:27.895132    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx" event={"ID":"c3cd58b7-297c-4b14-b353-e029001f2797","Type":"ContainerStarted","Data":"506e5d20c41b482184a71274ae298230c380ae2f4a34c938b0abb2209850561e"}
Sep 09 16:06:27.895252 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:27.895192    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx" event={"ID":"c3cd58b7-297c-4b14-b353-e029001f2797","Type":"ContainerStarted","Data":"b11f26955232698a81a76868bde1be3eaa4e0e6d8fa059ff1ad7901dc562960f"}
Sep 09 16:06:27.910960 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:27.910874    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx" podStartSLOduration=1.661042798 podStartE2EDuration="1.910851929s" totalImagesPullingTime="249.809131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:06:27.910626116 +0000 UTC m=+892.777137338" watchObservedRunningTime="2026-09-09 16:06:27.910851929 +0000 UTC m=+892.777363150"
Sep 09 16:06:33.844401 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:33.844353    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:06:35.619800 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:35.619765    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 16:06:35.620482 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:35.620445    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 16:06:36.967810 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:36.967774    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:36.970175 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:36.967852    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:36.980578 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:36.980555    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:37.947673 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:37.947642    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pppdx"
Sep 09 16:06:57.250618 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:06:57.250573    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:07:02.271352 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.271320    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"]
Sep 09 16:07:02.274291 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.274272    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.277043 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.277025    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 09 16:07:02.294424 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.294389    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"]
Sep 09 16:07:02.318993 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.318957    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:07:02.333576 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.333549    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9de4b71-89ec-4a22-8402-3054f4411a5c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.333709 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.333601    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqjjn\" (UniqueName: \"kubernetes.io/projected/e9de4b71-89ec-4a22-8402-3054f4411a5c-kube-api-access-zqjjn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.333709 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.333679    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9de4b71-89ec-4a22-8402-3054f4411a5c-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.333709 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.333704    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9de4b71-89ec-4a22-8402-3054f4411a5c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.333819 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.333731    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9de4b71-89ec-4a22-8402-3054f4411a5c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.333819 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.333762    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9de4b71-89ec-4a22-8402-3054f4411a5c-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.434939 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.434898    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9de4b71-89ec-4a22-8402-3054f4411a5c-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.434939 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.434941    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9de4b71-89ec-4a22-8402-3054f4411a5c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.435134 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.434986    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqjjn\" (UniqueName: \"kubernetes.io/projected/e9de4b71-89ec-4a22-8402-3054f4411a5c-kube-api-access-zqjjn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.435134 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.435015    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9de4b71-89ec-4a22-8402-3054f4411a5c-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.435134 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.435037    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9de4b71-89ec-4a22-8402-3054f4411a5c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.435134 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.435069    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9de4b71-89ec-4a22-8402-3054f4411a5c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.435359 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.435331    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9de4b71-89ec-4a22-8402-3054f4411a5c-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.435489 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.435444    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9de4b71-89ec-4a22-8402-3054f4411a5c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.435566 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.435517    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9de4b71-89ec-4a22-8402-3054f4411a5c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.437250 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.437229    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9de4b71-89ec-4a22-8402-3054f4411a5c-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.437553 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.437537    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9de4b71-89ec-4a22-8402-3054f4411a5c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.444133 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.444109    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqjjn\" (UniqueName: \"kubernetes.io/projected/e9de4b71-89ec-4a22-8402-3054f4411a5c-kube-api-access-zqjjn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48ghc\" (UID: \"e9de4b71-89ec-4a22-8402-3054f4411a5c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.584318 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.584227    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:02.851913 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:02.851835    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"]
Sep 09 16:07:02.855309 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:07:02.855279    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9de4b71_89ec_4a22_8402_3054f4411a5c.slice/crio-df04f1f0cdcd7da17cc511de451c020a4ad017f3aae63e09a098bce41e19464c WatchSource:0}: Error finding container df04f1f0cdcd7da17cc511de451c020a4ad017f3aae63e09a098bce41e19464c: Status 404 returned error can't find the container with id df04f1f0cdcd7da17cc511de451c020a4ad017f3aae63e09a098bce41e19464c
Sep 09 16:07:03.041761 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:03.041710    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc" event={"ID":"e9de4b71-89ec-4a22-8402-3054f4411a5c","Type":"ContainerStarted","Data":"df04f1f0cdcd7da17cc511de451c020a4ad017f3aae63e09a098bce41e19464c"}
Sep 09 16:07:04.047809 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:04.047768    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc" event={"ID":"e9de4b71-89ec-4a22-8402-3054f4411a5c","Type":"ContainerStarted","Data":"4b0b1e746ac5aef584b002d93c780dcf107691688d517f6858766a0afb6989ce"}
Sep 09 16:07:04.065433 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:04.065375    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc" podStartSLOduration=1.8802755979999999 podStartE2EDuration="2.065358619s" totalImagesPullingTime="185.083021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:07:04.063479182 +0000 UTC m=+928.929990393" watchObservedRunningTime="2026-09-09 16:07:04.065358619 +0000 UTC m=+928.931869844"
Sep 09 16:07:07.869359 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:07.869329    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:07:12.584825 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:12.584790    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:12.585265 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:12.584900    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:12.597758 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:12.597730    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:13.097817 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:13.097789    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48ghc"
Sep 09 16:07:16.868182 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:16.868144    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:07:21.364449 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:07:21.364413    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:08:04.151299 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:04.151211    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:08:12.604975 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:12.604945    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:08:12.705324 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:12.705287    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:08:37.149534 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:37.149498    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:08:46.386855 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:46.386823    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6574f9ffbb-tt6rp"]
Sep 09 16:08:46.393809 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:46.393782    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6574f9ffbb-tt6rp"
Sep 09 16:08:46.397972 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:46.397942    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6574f9ffbb-tt6rp"]
Sep 09 16:08:46.486243 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:46.486210    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwqlm\" (UniqueName: \"kubernetes.io/projected/56f9152d-41ed-4827-8f66-c3b2b9af5c15-kube-api-access-kwqlm\") pod \"authorino-6574f9ffbb-tt6rp\" (UID: \"56f9152d-41ed-4827-8f66-c3b2b9af5c15\") " pod="kuadrant-system/authorino-6574f9ffbb-tt6rp"
Sep 09 16:08:46.486405 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:46.486267    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/56f9152d-41ed-4827-8f66-c3b2b9af5c15-tls-cert\") pod \"authorino-6574f9ffbb-tt6rp\" (UID: \"56f9152d-41ed-4827-8f66-c3b2b9af5c15\") " pod="kuadrant-system/authorino-6574f9ffbb-tt6rp"
Sep 09 16:08:46.586852 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:46.586818    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwqlm\" (UniqueName: \"kubernetes.io/projected/56f9152d-41ed-4827-8f66-c3b2b9af5c15-kube-api-access-kwqlm\") pod \"authorino-6574f9ffbb-tt6rp\" (UID: \"56f9152d-41ed-4827-8f66-c3b2b9af5c15\") " pod="kuadrant-system/authorino-6574f9ffbb-tt6rp"
Sep 09 16:08:46.587046 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:46.586885    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/56f9152d-41ed-4827-8f66-c3b2b9af5c15-tls-cert\") pod \"authorino-6574f9ffbb-tt6rp\" (UID: \"56f9152d-41ed-4827-8f66-c3b2b9af5c15\") " pod="kuadrant-system/authorino-6574f9ffbb-tt6rp"
Sep 09 16:08:46.589403 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:46.589382    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/56f9152d-41ed-4827-8f66-c3b2b9af5c15-tls-cert\") pod \"authorino-6574f9ffbb-tt6rp\" (UID: \"56f9152d-41ed-4827-8f66-c3b2b9af5c15\") " pod="kuadrant-system/authorino-6574f9ffbb-tt6rp"
Sep 09 16:08:46.597073 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:46.597044    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwqlm\" (UniqueName: \"kubernetes.io/projected/56f9152d-41ed-4827-8f66-c3b2b9af5c15-kube-api-access-kwqlm\") pod \"authorino-6574f9ffbb-tt6rp\" (UID: \"56f9152d-41ed-4827-8f66-c3b2b9af5c15\") " pod="kuadrant-system/authorino-6574f9ffbb-tt6rp"
Sep 09 16:08:46.703951 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:46.703918    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6574f9ffbb-tt6rp"
Sep 09 16:08:46.971530 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:46.971434    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6574f9ffbb-tt6rp"]
Sep 09 16:08:46.974533 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:08:46.974503    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56f9152d_41ed_4827_8f66_c3b2b9af5c15.slice/crio-3b98a1585b1bd1d54a07cb9fdcdefabdd0216398a6f13d18f476f68d3756a8a3 WatchSource:0}: Error finding container 3b98a1585b1bd1d54a07cb9fdcdefabdd0216398a6f13d18f476f68d3756a8a3: Status 404 returned error can't find the container with id 3b98a1585b1bd1d54a07cb9fdcdefabdd0216398a6f13d18f476f68d3756a8a3
Sep 09 16:08:46.975878 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:46.975857    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:08:47.469910 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:47.469875    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6574f9ffbb-tt6rp" event={"ID":"56f9152d-41ed-4827-8f66-c3b2b9af5c15","Type":"ContainerStarted","Data":"3b98a1585b1bd1d54a07cb9fdcdefabdd0216398a6f13d18f476f68d3756a8a3"}
Sep 09 16:08:48.477763 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:48.477718    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6574f9ffbb-tt6rp" event={"ID":"56f9152d-41ed-4827-8f66-c3b2b9af5c15","Type":"ContainerStarted","Data":"671397dd77c0a0e99200858c8a7509dd9b46b9ea3801054630fb623350231ba9"}
Sep 09 16:08:48.494504 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:48.494431    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6574f9ffbb-tt6rp" podStartSLOduration=1.578287013 podStartE2EDuration="2.494412797s" totalImagesPullingTime="916.125784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:08:48.492192543 +0000 UTC m=+1033.358703768" watchObservedRunningTime="2026-09-09 16:08:48.494412797 +0000 UTC m=+1033.360924017"
Sep 09 16:08:48.519616 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:48.519581    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-647d78969f-f5mzk"]
Sep 09 16:08:48.519808 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:48.519758    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-647d78969f-f5mzk" podUID="77e64e31-d52e-4e26-985a-1ec780faebfb" containerName="authorino" containerID="cri-o://4614950a9c021c4d26e1300e1a453177abddd983733f2a2e30a798af594c296d" gracePeriod=30
Sep 09 16:08:48.762898 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:48.762872    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-647d78969f-f5mzk"
Sep 09 16:08:48.907896 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:48.907864    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-699b2\" (UniqueName: \"kubernetes.io/projected/77e64e31-d52e-4e26-985a-1ec780faebfb-kube-api-access-699b2\") pod \"77e64e31-d52e-4e26-985a-1ec780faebfb\" (UID: \"77e64e31-d52e-4e26-985a-1ec780faebfb\") "
Sep 09 16:08:48.908059 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:48.907923    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77e64e31-d52e-4e26-985a-1ec780faebfb-tls-cert\") pod \"77e64e31-d52e-4e26-985a-1ec780faebfb\" (UID: \"77e64e31-d52e-4e26-985a-1ec780faebfb\") "
Sep 09 16:08:48.910605 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:48.910579    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e64e31-d52e-4e26-985a-1ec780faebfb-kube-api-access-699b2" (OuterVolumeSpecName: "kube-api-access-699b2") pod "77e64e31-d52e-4e26-985a-1ec780faebfb" (UID: "77e64e31-d52e-4e26-985a-1ec780faebfb"). InnerVolumeSpecName "kube-api-access-699b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:08:48.918134 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:48.918112    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e64e31-d52e-4e26-985a-1ec780faebfb-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "77e64e31-d52e-4e26-985a-1ec780faebfb" (UID: "77e64e31-d52e-4e26-985a-1ec780faebfb"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:08:49.009699 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:49.009613    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-699b2\" (UniqueName: \"kubernetes.io/projected/77e64e31-d52e-4e26-985a-1ec780faebfb-kube-api-access-699b2\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:08:49.009699 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:49.009646    2556 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77e64e31-d52e-4e26-985a-1ec780faebfb-tls-cert\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:08:49.080362 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:49.080320    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:08:49.179648 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:49.179607    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:08:49.485612 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:49.485578    2556 generic.go:356] "Generic (PLEG): container finished" podID="77e64e31-d52e-4e26-985a-1ec780faebfb" containerID="4614950a9c021c4d26e1300e1a453177abddd983733f2a2e30a798af594c296d" exitCode=0
Sep 09 16:08:49.486077 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:49.485622    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-647d78969f-f5mzk"
Sep 09 16:08:49.486077 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:49.485662    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-647d78969f-f5mzk" event={"ID":"77e64e31-d52e-4e26-985a-1ec780faebfb","Type":"ContainerDied","Data":"4614950a9c021c4d26e1300e1a453177abddd983733f2a2e30a798af594c296d"}
Sep 09 16:08:49.486077 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:49.485715    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-647d78969f-f5mzk" event={"ID":"77e64e31-d52e-4e26-985a-1ec780faebfb","Type":"ContainerDied","Data":"307f966c29925f9fe59c4780a9f08f34ab7a619b064609a0126ccaef952bb1a6"}
Sep 09 16:08:49.486077 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:49.485739    2556 scope.go:117] "RemoveContainer" containerID="4614950a9c021c4d26e1300e1a453177abddd983733f2a2e30a798af594c296d"
Sep 09 16:08:49.496236 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:49.496216    2556 scope.go:117] "RemoveContainer" containerID="4614950a9c021c4d26e1300e1a453177abddd983733f2a2e30a798af594c296d"
Sep 09 16:08:49.496572 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:08:49.496553    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4614950a9c021c4d26e1300e1a453177abddd983733f2a2e30a798af594c296d\": container with ID starting with 4614950a9c021c4d26e1300e1a453177abddd983733f2a2e30a798af594c296d not found: ID does not exist" containerID="4614950a9c021c4d26e1300e1a453177abddd983733f2a2e30a798af594c296d"
Sep 09 16:08:49.496654 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:49.496583    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4614950a9c021c4d26e1300e1a453177abddd983733f2a2e30a798af594c296d"} err="failed to get container status \"4614950a9c021c4d26e1300e1a453177abddd983733f2a2e30a798af594c296d\": rpc error: code = NotFound desc = could not find container \"4614950a9c021c4d26e1300e1a453177abddd983733f2a2e30a798af594c296d\": container with ID starting with 4614950a9c021c4d26e1300e1a453177abddd983733f2a2e30a798af594c296d not found: ID does not exist"
Sep 09 16:08:49.506680 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:49.506655    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-647d78969f-f5mzk"]
Sep 09 16:08:49.511273 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:49.511251    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-647d78969f-f5mzk"]
Sep 09 16:08:49.762936 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:49.762861    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77e64e31-d52e-4e26-985a-1ec780faebfb" path="/var/lib/kubelet/pods/77e64e31-d52e-4e26-985a-1ec780faebfb/volumes"
Sep 09 16:08:50.279722 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:50.279680    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:08:50.373305 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:08:50.373273    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:07.814200 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:07.814158    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:09.775678 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:09.775643    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:09.887202 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:09.887171    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:09.987715 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:09.987687    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:10.017530 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:10.017492    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7707-89528d"]
Sep 09 16:09:10.017828 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:10.017812    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e64e31-d52e-4e26-985a-1ec780faebfb" containerName="authorino"
Sep 09 16:09:10.017870 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:10.017830    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e64e31-d52e-4e26-985a-1ec780faebfb" containerName="authorino"
Sep 09 16:09:10.017904 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:10.017885    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="77e64e31-d52e-4e26-985a-1ec780faebfb" containerName="authorino"
Sep 09 16:09:10.020671 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:10.020654    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-89528d"
Sep 09 16:09:10.022944 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:10.022926    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lp4n9\""
Sep 09 16:09:10.028454 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:10.028432    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-89528d"]
Sep 09 16:09:10.079834 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:10.079792    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqscz\" (UniqueName: \"kubernetes.io/projected/a09b3f29-1780-40ce-b7bb-e2059e7492b4-kube-api-access-qqscz\") pod \"test-curl-7707-89528d\" (UID: \"a09b3f29-1780-40ce-b7bb-e2059e7492b4\") " pod="opendatahub/test-curl-7707-89528d"
Sep 09 16:09:10.181142 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:10.181105    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqscz\" (UniqueName: \"kubernetes.io/projected/a09b3f29-1780-40ce-b7bb-e2059e7492b4-kube-api-access-qqscz\") pod \"test-curl-7707-89528d\" (UID: \"a09b3f29-1780-40ce-b7bb-e2059e7492b4\") " pod="opendatahub/test-curl-7707-89528d"
Sep 09 16:09:10.188345 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:10.188308    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqscz\" (UniqueName: \"kubernetes.io/projected/a09b3f29-1780-40ce-b7bb-e2059e7492b4-kube-api-access-qqscz\") pod \"test-curl-7707-89528d\" (UID: \"a09b3f29-1780-40ce-b7bb-e2059e7492b4\") " pod="opendatahub/test-curl-7707-89528d"
Sep 09 16:09:10.330523 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:10.330418    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-89528d"
Sep 09 16:09:10.597523 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:10.597378    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-89528d"]
Sep 09 16:09:10.851617 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:10.851527    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:10.955350 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:10.955315    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:11.579297 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:11.579263    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-89528d" event={"ID":"a09b3f29-1780-40ce-b7bb-e2059e7492b4","Type":"ContainerStarted","Data":"8e1b46d30927e091c30bee3906b0f64f10c978a6868c171a1bde580b15711488"}
Sep 09 16:09:12.395115 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:12.395062    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:12.501313 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:12.501274    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:12.695865 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:12.695831    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:12.855782 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:12.855745    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"]
Sep 09 16:09:12.859205 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:12.859184    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"
Sep 09 16:09:12.861627 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:12.861603    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7b8369-dockercfg-wcb4x\""
Sep 09 16:09:12.861744 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:12.861632    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7b8369\""
Sep 09 16:09:12.871178 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:12.871152    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"]
Sep 09 16:09:12.903905 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:12.903868    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrt6s\" (UniqueName: \"kubernetes.io/projected/8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee-kube-api-access-jrt6s\") pod \"payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww\" (UID: \"8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"
Sep 09 16:09:12.904085 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:12.903943    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww\" (UID: \"8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"
Sep 09 16:09:12.986419 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:12.986329    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:13.004527 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:13.004497    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrt6s\" (UniqueName: \"kubernetes.io/projected/8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee-kube-api-access-jrt6s\") pod \"payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww\" (UID: \"8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"
Sep 09 16:09:13.004703 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:13.004579    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww\" (UID: \"8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"
Sep 09 16:09:13.005252 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:13.005227    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww\" (UID: \"8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"
Sep 09 16:09:13.013230 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:13.013207    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrt6s\" (UniqueName: \"kubernetes.io/projected/8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee-kube-api-access-jrt6s\") pod \"payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww\" (UID: \"8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"
Sep 09 16:09:13.086710 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:13.086669    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:13.170234 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:13.170200    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"
Sep 09 16:09:13.436633 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:13.436610    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"]
Sep 09 16:09:13.438610 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:09:13.438578    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec0588b_dcf1_4cfa_93ef_0f865d8bd7ee.slice/crio-7e24bf666c0c8e76f32218cd8beb847d4529201e78ace58ce3baf1b35e0062ee WatchSource:0}: Error finding container 7e24bf666c0c8e76f32218cd8beb847d4529201e78ace58ce3baf1b35e0062ee: Status 404 returned error can't find the container with id 7e24bf666c0c8e76f32218cd8beb847d4529201e78ace58ce3baf1b35e0062ee
Sep 09 16:09:13.590447 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:13.590405    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-89528d" event={"ID":"a09b3f29-1780-40ce-b7bb-e2059e7492b4","Type":"ContainerStarted","Data":"16f3bc4a29ad35b6f4be82144e6608dcf9782832df3850ce81a56105ea174162"}
Sep 09 16:09:13.591982 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:13.591957    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww" event={"ID":"8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee","Type":"ContainerStarted","Data":"545bd91c14692503e44c400d4f05ff2894003a9a78ffdfecdf0199e097da0249"}
Sep 09 16:09:13.592117 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:13.592003    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww" event={"ID":"8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee","Type":"ContainerStarted","Data":"7e24bf666c0c8e76f32218cd8beb847d4529201e78ace58ce3baf1b35e0062ee"}
Sep 09 16:09:13.592117 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:13.592044    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"
Sep 09 16:09:13.601617 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:13.601564    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7707-89528d" podStartSLOduration=1.425536959 podStartE2EDuration="3.601543718s" totalImagesPullingTime="2.176006759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:13.600877965 +0000 UTC m=+1058.467389183" watchObservedRunningTime="2026-09-09 16:09:13.601543718 +0000 UTC m=+1058.468054956"
Sep 09 16:09:13.615208 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:13.615165    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww" podStartSLOduration=1.615149868 podStartE2EDuration="1.615149868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:13.613592262 +0000 UTC m=+1058.480103482" watchObservedRunningTime="2026-09-09 16:09:13.615149868 +0000 UTC m=+1058.481661088"
Sep 09 16:09:14.201503 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:14.201455    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7707-89528d"]
Sep 09 16:09:14.204397 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:14.204367    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7707-89528d"]
Sep 09 16:09:14.324857 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:14.324822    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7707-14fa38"]
Sep 09 16:09:14.325237 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:14.325223    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09b3f29-1780-40ce-b7bb-e2059e7492b4" containerName="test-curl-7707-89528d"
Sep 09 16:09:14.325286 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:14.325241    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09b3f29-1780-40ce-b7bb-e2059e7492b4" containerName="test-curl-7707-89528d"
Sep 09 16:09:14.325348 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:14.325338    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a09b3f29-1780-40ce-b7bb-e2059e7492b4" containerName="test-curl-7707-89528d"
Sep 09 16:09:14.328600 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:14.328580    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-14fa38"
Sep 09 16:09:14.331000 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:14.330973    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-14fa38"]
Sep 09 16:09:14.418543 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:14.418505    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w878h\" (UniqueName: \"kubernetes.io/projected/f248e86c-7b89-42a7-b626-46a4537a8d50-kube-api-access-w878h\") pod \"test-curl-7707-14fa38\" (UID: \"f248e86c-7b89-42a7-b626-46a4537a8d50\") " pod="opendatahub/test-curl-7707-14fa38"
Sep 09 16:09:14.519910 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:14.519831    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w878h\" (UniqueName: \"kubernetes.io/projected/f248e86c-7b89-42a7-b626-46a4537a8d50-kube-api-access-w878h\") pod \"test-curl-7707-14fa38\" (UID: \"f248e86c-7b89-42a7-b626-46a4537a8d50\") " pod="opendatahub/test-curl-7707-14fa38"
Sep 09 16:09:14.528062 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:14.528040    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w878h\" (UniqueName: \"kubernetes.io/projected/f248e86c-7b89-42a7-b626-46a4537a8d50-kube-api-access-w878h\") pod \"test-curl-7707-14fa38\" (UID: \"f248e86c-7b89-42a7-b626-46a4537a8d50\") " pod="opendatahub/test-curl-7707-14fa38"
Sep 09 16:09:14.596786 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:09:14.596757    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7707-89528d" container="test-curl-7707-89528d"
Sep 09 16:09:14.640332 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:14.640294    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-14fa38"
Sep 09 16:09:14.905903 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:14.905881    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-14fa38"]
Sep 09 16:09:14.908868 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:09:14.908836    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf248e86c_7b89_42a7_b626_46a4537a8d50.slice/crio-4ad020fe3968e791a0fdae11e1835bc85483c70a4c7615edc9189ed7751a5ad4 WatchSource:0}: Error finding container 4ad020fe3968e791a0fdae11e1835bc85483c70a4c7615edc9189ed7751a5ad4: Status 404 returned error can't find the container with id 4ad020fe3968e791a0fdae11e1835bc85483c70a4c7615edc9189ed7751a5ad4
Sep 09 16:09:15.603432 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:15.603394    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-14fa38" event={"ID":"f248e86c-7b89-42a7-b626-46a4537a8d50","Type":"ContainerStarted","Data":"6ac43b61c73171ea72ba35a8e03a3e6d34652d7ada87443952fdbf38f250ed25"}
Sep 09 16:09:15.603432 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:15.603443    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-14fa38" event={"ID":"f248e86c-7b89-42a7-b626-46a4537a8d50","Type":"ContainerStarted","Data":"4ad020fe3968e791a0fdae11e1835bc85483c70a4c7615edc9189ed7751a5ad4"}
Sep 09 16:09:15.603941 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:15.603450    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7707-89528d" podUID="a09b3f29-1780-40ce-b7bb-e2059e7492b4" containerName="test-curl-7707-89528d" containerID="cri-o://16f3bc4a29ad35b6f4be82144e6608dcf9782832df3850ce81a56105ea174162" gracePeriod=2
Sep 09 16:09:15.605540 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:09:15.605505    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7707-89528d\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="a09b3f29-1780-40ce-b7bb-e2059e7492b4" pod="opendatahub/test-curl-7707-89528d"
Sep 09 16:09:15.615738 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:15.615688    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7707-14fa38" podStartSLOduration=1.615673076 podStartE2EDuration="1.615673076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:15.61551047 +0000 UTC m=+1060.482021707" watchObservedRunningTime="2026-09-09 16:09:15.615673076 +0000 UTC m=+1060.482184306"
Sep 09 16:09:15.617220 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:09:15.617196    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7707-89528d\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="a09b3f29-1780-40ce-b7bb-e2059e7492b4" pod="opendatahub/test-curl-7707-89528d"
Sep 09 16:09:15.761287 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:09:15.761256    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7707-89528d\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="a09b3f29-1780-40ce-b7bb-e2059e7492b4" pod="opendatahub/test-curl-7707-89528d"
Sep 09 16:09:15.767233 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:15.767182    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:15.825592 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:15.825552    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:16.018293 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:16.018255    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:16.262010 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:16.261962    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7707-14fa38"]
Sep 09 16:09:16.266526 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:16.266422    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7707-14fa38"]
Sep 09 16:09:16.349780 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:16.349742    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:16.479389 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:16.479351    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:16.608948 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:09:16.608876    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7707-14fa38" container="test-curl-7707-14fa38"
Sep 09 16:09:16.806576 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:16.806541    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:17.088682 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:17.088638    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:17.233433 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:17.233395    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:17.613352 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:17.613318    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7707-14fa38" podUID="f248e86c-7b89-42a7-b626-46a4537a8d50" containerName="test-curl-7707-14fa38" containerID="cri-o://6ac43b61c73171ea72ba35a8e03a3e6d34652d7ada87443952fdbf38f250ed25" gracePeriod=2
Sep 09 16:09:17.615334 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:09:17.615305    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7707-14fa38\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="f248e86c-7b89-42a7-b626-46a4537a8d50" pod="opendatahub/test-curl-7707-14fa38"
Sep 09 16:09:17.790708 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:17.790685    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-89528d"
Sep 09 16:09:17.846921 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:17.846890    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqscz\" (UniqueName: \"kubernetes.io/projected/a09b3f29-1780-40ce-b7bb-e2059e7492b4-kube-api-access-qqscz\") pod \"a09b3f29-1780-40ce-b7bb-e2059e7492b4\" (UID: \"a09b3f29-1780-40ce-b7bb-e2059e7492b4\") "
Sep 09 16:09:17.849040 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:17.849010    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09b3f29-1780-40ce-b7bb-e2059e7492b4-kube-api-access-qqscz" (OuterVolumeSpecName: "kube-api-access-qqscz") pod "a09b3f29-1780-40ce-b7bb-e2059e7492b4" (UID: "a09b3f29-1780-40ce-b7bb-e2059e7492b4"). InnerVolumeSpecName "kube-api-access-qqscz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:09:17.948712 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:17.948676    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqscz\" (UniqueName: \"kubernetes.io/projected/a09b3f29-1780-40ce-b7bb-e2059e7492b4-kube-api-access-qqscz\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:09:18.619334 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:18.619301    2556 generic.go:356] "Generic (PLEG): container finished" podID="a09b3f29-1780-40ce-b7bb-e2059e7492b4" containerID="16f3bc4a29ad35b6f4be82144e6608dcf9782832df3850ce81a56105ea174162" exitCode=137
Sep 09 16:09:18.619748 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:18.619345    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-89528d"
Sep 09 16:09:18.619748 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:18.619350    2556 scope.go:117] "RemoveContainer" containerID="16f3bc4a29ad35b6f4be82144e6608dcf9782832df3850ce81a56105ea174162"
Sep 09 16:09:18.627490 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:18.627475    2556 scope.go:117] "RemoveContainer" containerID="16f3bc4a29ad35b6f4be82144e6608dcf9782832df3850ce81a56105ea174162"
Sep 09 16:09:18.627757 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:09:18.627741    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f3bc4a29ad35b6f4be82144e6608dcf9782832df3850ce81a56105ea174162\": container with ID starting with 16f3bc4a29ad35b6f4be82144e6608dcf9782832df3850ce81a56105ea174162 not found: ID does not exist" containerID="16f3bc4a29ad35b6f4be82144e6608dcf9782832df3850ce81a56105ea174162"
Sep 09 16:09:18.627802 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:18.627764    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f3bc4a29ad35b6f4be82144e6608dcf9782832df3850ce81a56105ea174162"} err="failed to get container status \"16f3bc4a29ad35b6f4be82144e6608dcf9782832df3850ce81a56105ea174162\": rpc error: code = NotFound desc = could not find container \"16f3bc4a29ad35b6f4be82144e6608dcf9782832df3850ce81a56105ea174162\": container with ID starting with 16f3bc4a29ad35b6f4be82144e6608dcf9782832df3850ce81a56105ea174162 not found: ID does not exist"
Sep 09 16:09:18.633226 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:09:18.633208    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7707-89528d\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="a09b3f29-1780-40ce-b7bb-e2059e7492b4" pod="opendatahub/test-curl-7707-89528d"
Sep 09 16:09:19.310695 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:19.310656    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:19.415922 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:19.415887    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:19.764327 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:19.764292    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a09b3f29-1780-40ce-b7bb-e2059e7492b4" path="/var/lib/kubelet/pods/a09b3f29-1780-40ce-b7bb-e2059e7492b4/volumes"
Sep 09 16:09:19.786974 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:19.786951    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-14fa38"
Sep 09 16:09:19.865247 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:19.865183    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w878h\" (UniqueName: \"kubernetes.io/projected/f248e86c-7b89-42a7-b626-46a4537a8d50-kube-api-access-w878h\") pod \"f248e86c-7b89-42a7-b626-46a4537a8d50\" (UID: \"f248e86c-7b89-42a7-b626-46a4537a8d50\") "
Sep 09 16:09:19.867498 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:19.867423    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f248e86c-7b89-42a7-b626-46a4537a8d50-kube-api-access-w878h" (OuterVolumeSpecName: "kube-api-access-w878h") pod "f248e86c-7b89-42a7-b626-46a4537a8d50" (UID: "f248e86c-7b89-42a7-b626-46a4537a8d50"). InnerVolumeSpecName "kube-api-access-w878h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:09:19.900010 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:19.899982    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:19.966330 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:19.966302    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w878h\" (UniqueName: \"kubernetes.io/projected/f248e86c-7b89-42a7-b626-46a4537a8d50-kube-api-access-w878h\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:09:19.999871 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:19.999826    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:20.475011 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:20.474975    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:20.612634 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:20.612597    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:20.630849 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:20.630819    2556 generic.go:356] "Generic (PLEG): container finished" podID="f248e86c-7b89-42a7-b626-46a4537a8d50" containerID="6ac43b61c73171ea72ba35a8e03a3e6d34652d7ada87443952fdbf38f250ed25" exitCode=137
Sep 09 16:09:20.630999 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:20.630868    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-14fa38"
Sep 09 16:09:20.630999 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:20.630891    2556 scope.go:117] "RemoveContainer" containerID="6ac43b61c73171ea72ba35a8e03a3e6d34652d7ada87443952fdbf38f250ed25"
Sep 09 16:09:20.638787 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:20.638767    2556 scope.go:117] "RemoveContainer" containerID="6ac43b61c73171ea72ba35a8e03a3e6d34652d7ada87443952fdbf38f250ed25"
Sep 09 16:09:20.639066 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:09:20.639045    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac43b61c73171ea72ba35a8e03a3e6d34652d7ada87443952fdbf38f250ed25\": container with ID starting with 6ac43b61c73171ea72ba35a8e03a3e6d34652d7ada87443952fdbf38f250ed25 not found: ID does not exist" containerID="6ac43b61c73171ea72ba35a8e03a3e6d34652d7ada87443952fdbf38f250ed25"
Sep 09 16:09:20.639114 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:20.639074    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac43b61c73171ea72ba35a8e03a3e6d34652d7ada87443952fdbf38f250ed25"} err="failed to get container status \"6ac43b61c73171ea72ba35a8e03a3e6d34652d7ada87443952fdbf38f250ed25\": rpc error: code = NotFound desc = could not find container \"6ac43b61c73171ea72ba35a8e03a3e6d34652d7ada87443952fdbf38f250ed25\": container with ID starting with 6ac43b61c73171ea72ba35a8e03a3e6d34652d7ada87443952fdbf38f250ed25 not found: ID does not exist"
Sep 09 16:09:20.643277 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:09:20.643252    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7707-14fa38\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="f248e86c-7b89-42a7-b626-46a4537a8d50" pod="opendatahub/test-curl-7707-14fa38"
Sep 09 16:09:21.000234 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.000193    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:21.098681 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.098634    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:21.146118 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.146085    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"]
Sep 09 16:09:21.146508 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.146491    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f248e86c-7b89-42a7-b626-46a4537a8d50" containerName="test-curl-7707-14fa38"
Sep 09 16:09:21.146508 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.146509    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f248e86c-7b89-42a7-b626-46a4537a8d50" containerName="test-curl-7707-14fa38"
Sep 09 16:09:21.146670 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.146579    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f248e86c-7b89-42a7-b626-46a4537a8d50" containerName="test-curl-7707-14fa38"
Sep 09 16:09:21.150746 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.150725    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"
Sep 09 16:09:21.153126 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:09:21.153097    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7707-14fa38\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="f248e86c-7b89-42a7-b626-46a4537a8d50" pod="opendatahub/test-curl-7707-14fa38"
Sep 09 16:09:21.153246 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.153134    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-083463c6\""
Sep 09 16:09:21.161336 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.161305    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"]
Sep 09 16:09:21.174234 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.174204    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"]
Sep 09 16:09:21.176605 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.176577    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/518bb67d-c896-4cb7-bebb-8bddd100b7df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427\" (UID: \"518bb67d-c896-4cb7-bebb-8bddd100b7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"
Sep 09 16:09:21.176736 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.176635    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t6r9\" (UniqueName: \"kubernetes.io/projected/518bb67d-c896-4cb7-bebb-8bddd100b7df-kube-api-access-4t6r9\") pod \"payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427\" (UID: \"518bb67d-c896-4cb7-bebb-8bddd100b7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"
Sep 09 16:09:21.177451 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.177425    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:21.177842 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.177825    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"
Sep 09 16:09:21.180498 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.180479    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-083463c6-dockercfg-tzm2l\""
Sep 09 16:09:21.187079 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.187057    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"]
Sep 09 16:09:21.202794 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:09:21.202768    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7707-14fa38\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="f248e86c-7b89-42a7-b626-46a4537a8d50" pod="opendatahub/test-curl-7707-14fa38"
Sep 09 16:09:21.277429 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.277358    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/518bb67d-c896-4cb7-bebb-8bddd100b7df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427\" (UID: \"518bb67d-c896-4cb7-bebb-8bddd100b7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"
Sep 09 16:09:21.277585 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.277425    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t6r9\" (UniqueName: \"kubernetes.io/projected/518bb67d-c896-4cb7-bebb-8bddd100b7df-kube-api-access-4t6r9\") pod \"payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427\" (UID: \"518bb67d-c896-4cb7-bebb-8bddd100b7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"
Sep 09 16:09:21.277585 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.277460    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn2b9\" (UniqueName: \"kubernetes.io/projected/68e29f7c-6c42-4630-90f7-ef961ba55b5c-kube-api-access-cn2b9\") pod \"payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq\" (UID: \"68e29f7c-6c42-4630-90f7-ef961ba55b5c\") " pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"
Sep 09 16:09:21.277585 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.277525    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e29f7c-6c42-4630-90f7-ef961ba55b5c-plugins-config-volume\") pod \"payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq\" (UID: \"68e29f7c-6c42-4630-90f7-ef961ba55b5c\") " pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"
Sep 09 16:09:21.278083 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.278063    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/518bb67d-c896-4cb7-bebb-8bddd100b7df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427\" (UID: \"518bb67d-c896-4cb7-bebb-8bddd100b7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"
Sep 09 16:09:21.287709 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.287687    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t6r9\" (UniqueName: \"kubernetes.io/projected/518bb67d-c896-4cb7-bebb-8bddd100b7df-kube-api-access-4t6r9\") pod \"payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427\" (UID: \"518bb67d-c896-4cb7-bebb-8bddd100b7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"
Sep 09 16:09:21.378635 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.378604    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn2b9\" (UniqueName: \"kubernetes.io/projected/68e29f7c-6c42-4630-90f7-ef961ba55b5c-kube-api-access-cn2b9\") pod \"payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq\" (UID: \"68e29f7c-6c42-4630-90f7-ef961ba55b5c\") " pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"
Sep 09 16:09:21.378635 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.378641    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e29f7c-6c42-4630-90f7-ef961ba55b5c-plugins-config-volume\") pod \"payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq\" (UID: \"68e29f7c-6c42-4630-90f7-ef961ba55b5c\") " pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"
Sep 09 16:09:21.379301 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.379283    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e29f7c-6c42-4630-90f7-ef961ba55b5c-plugins-config-volume\") pod \"payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq\" (UID: \"68e29f7c-6c42-4630-90f7-ef961ba55b5c\") " pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"
Sep 09 16:09:21.386123 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.386103    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn2b9\" (UniqueName: \"kubernetes.io/projected/68e29f7c-6c42-4630-90f7-ef961ba55b5c-kube-api-access-cn2b9\") pod \"payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq\" (UID: \"68e29f7c-6c42-4630-90f7-ef961ba55b5c\") " pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"
Sep 09 16:09:21.461160 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.461129    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"
Sep 09 16:09:21.487589 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.487557    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"
Sep 09 16:09:21.566529 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.566179    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:21.627703 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.627668    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:21.732274 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.732249    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"]
Sep 09 16:09:21.734838 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:09:21.734809    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518bb67d_c896_4cb7_bebb_8bddd100b7df.slice/crio-425e3d6005358761adbcceecc35f842bf14670ef86958e5f527d3966b3dd7a1e WatchSource:0}: Error finding container 425e3d6005358761adbcceecc35f842bf14670ef86958e5f527d3966b3dd7a1e: Status 404 returned error can't find the container with id 425e3d6005358761adbcceecc35f842bf14670ef86958e5f527d3966b3dd7a1e
Sep 09 16:09:21.736911 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.736881    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:21.762393 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.762364    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f248e86c-7b89-42a7-b626-46a4537a8d50" path="/var/lib/kubelet/pods/f248e86c-7b89-42a7-b626-46a4537a8d50/volumes"
Sep 09 16:09:21.767245 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:21.767220    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"]
Sep 09 16:09:21.770059 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:09:21.770039    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e29f7c_6c42_4630_90f7_ef961ba55b5c.slice/crio-c29026f2d897ee3c75ecfdd023ba737e54df6e389c311aa20769b7b2549255f0 WatchSource:0}: Error finding container c29026f2d897ee3c75ecfdd023ba737e54df6e389c311aa20769b7b2549255f0: Status 404 returned error can't find the container with id c29026f2d897ee3c75ecfdd023ba737e54df6e389c311aa20769b7b2549255f0
Sep 09 16:09:22.145979 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.145893    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:22.244977 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.244932    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:22.252558 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.252530    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"]
Sep 09 16:09:22.256163 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.256145    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"
Sep 09 16:09:22.258276 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.258255    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-a09c2a\""
Sep 09 16:09:22.258397 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.258287    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-a09c2a-dockercfg-68fp2\""
Sep 09 16:09:22.266089 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.266060    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"]
Sep 09 16:09:22.284356 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.284325    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"]
Sep 09 16:09:22.287108 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.287086    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ttcv\" (UniqueName: \"kubernetes.io/projected/426dd61b-5ccc-4666-954c-05a6924e5524-kube-api-access-9ttcv\") pod \"payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt\" (UID: \"426dd61b-5ccc-4666-954c-05a6924e5524\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"
Sep 09 16:09:22.287231 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.287130    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426dd61b-5ccc-4666-954c-05a6924e5524-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt\" (UID: \"426dd61b-5ccc-4666-954c-05a6924e5524\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"
Sep 09 16:09:22.287514 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.287498    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"
Sep 09 16:09:22.297364 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.297337    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"]
Sep 09 16:09:22.388161 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.388131    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ttcv\" (UniqueName: \"kubernetes.io/projected/426dd61b-5ccc-4666-954c-05a6924e5524-kube-api-access-9ttcv\") pod \"payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt\" (UID: \"426dd61b-5ccc-4666-954c-05a6924e5524\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"
Sep 09 16:09:22.388340 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.388182    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426dd61b-5ccc-4666-954c-05a6924e5524-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt\" (UID: \"426dd61b-5ccc-4666-954c-05a6924e5524\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"
Sep 09 16:09:22.388340 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.388217    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf547411-e2ec-45ab-b55b-8d8930e411e9-plugins-config-volume\") pod \"payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm\" (UID: \"bf547411-e2ec-45ab-b55b-8d8930e411e9\") " pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"
Sep 09 16:09:22.388340 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.388290    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xssm8\" (UniqueName: \"kubernetes.io/projected/bf547411-e2ec-45ab-b55b-8d8930e411e9-kube-api-access-xssm8\") pod \"payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm\" (UID: \"bf547411-e2ec-45ab-b55b-8d8930e411e9\") " pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"
Sep 09 16:09:22.388969 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.388950    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426dd61b-5ccc-4666-954c-05a6924e5524-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt\" (UID: \"426dd61b-5ccc-4666-954c-05a6924e5524\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"
Sep 09 16:09:22.396452 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.396393    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ttcv\" (UniqueName: \"kubernetes.io/projected/426dd61b-5ccc-4666-954c-05a6924e5524-kube-api-access-9ttcv\") pod \"payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt\" (UID: \"426dd61b-5ccc-4666-954c-05a6924e5524\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"
Sep 09 16:09:22.489324 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.489285    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf547411-e2ec-45ab-b55b-8d8930e411e9-plugins-config-volume\") pod \"payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm\" (UID: \"bf547411-e2ec-45ab-b55b-8d8930e411e9\") " pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"
Sep 09 16:09:22.489533 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.489354    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xssm8\" (UniqueName: \"kubernetes.io/projected/bf547411-e2ec-45ab-b55b-8d8930e411e9-kube-api-access-xssm8\") pod \"payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm\" (UID: \"bf547411-e2ec-45ab-b55b-8d8930e411e9\") " pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"
Sep 09 16:09:22.489991 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.489970    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf547411-e2ec-45ab-b55b-8d8930e411e9-plugins-config-volume\") pod \"payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm\" (UID: \"bf547411-e2ec-45ab-b55b-8d8930e411e9\") " pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"
Sep 09 16:09:22.498838 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.498816    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xssm8\" (UniqueName: \"kubernetes.io/projected/bf547411-e2ec-45ab-b55b-8d8930e411e9-kube-api-access-xssm8\") pod \"payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm\" (UID: \"bf547411-e2ec-45ab-b55b-8d8930e411e9\") " pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"
Sep 09 16:09:22.569105 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.569072    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"
Sep 09 16:09:22.607609 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.607575    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"
Sep 09 16:09:22.643891 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.643780    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427" event={"ID":"518bb67d-c896-4cb7-bebb-8bddd100b7df","Type":"ContainerStarted","Data":"00c7289ca2c74707058cc241a52ed50e9ac7c1406dbfbfeb01ded18892b66250"}
Sep 09 16:09:22.643891 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.643853    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427" event={"ID":"518bb67d-c896-4cb7-bebb-8bddd100b7df","Type":"ContainerStarted","Data":"425e3d6005358761adbcceecc35f842bf14670ef86958e5f527d3966b3dd7a1e"}
Sep 09 16:09:22.643891 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.643886    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"
Sep 09 16:09:22.645159 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.645128    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:22.645665 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.645642    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq" event={"ID":"68e29f7c-6c42-4630-90f7-ef961ba55b5c","Type":"ContainerStarted","Data":"184a50f55f64f93178e024fa1ff1a7e065f084a572fc41bdb7e7ee46a3127489"}
Sep 09 16:09:22.645773 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.645690    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq" event={"ID":"68e29f7c-6c42-4630-90f7-ef961ba55b5c","Type":"ContainerStarted","Data":"c29026f2d897ee3c75ecfdd023ba737e54df6e389c311aa20769b7b2549255f0"}
Sep 09 16:09:22.645773 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.645729    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"
Sep 09 16:09:22.659768 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.659653    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427" podStartSLOduration=1.659631148 podStartE2EDuration="1.659631148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:22.658427238 +0000 UTC m=+1067.524938469" watchObservedRunningTime="2026-09-09 16:09:22.659631148 +0000 UTC m=+1067.526142369"
Sep 09 16:09:22.846978 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.846920    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq" podStartSLOduration=1.846898739 podStartE2EDuration="1.846898739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:22.675643438 +0000 UTC m=+1067.542154659" watchObservedRunningTime="2026-09-09 16:09:22.846898739 +0000 UTC m=+1067.713409961"
Sep 09 16:09:22.847514 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.847495    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"]
Sep 09 16:09:22.849481 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:09:22.849444    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426dd61b_5ccc_4666_954c_05a6924e5524.slice/crio-c961bb28f855b20227752549aa28bc9152e5fad986d61dc1f83bf8a72af9be9c WatchSource:0}: Error finding container c961bb28f855b20227752549aa28bc9152e5fad986d61dc1f83bf8a72af9be9c: Status 404 returned error can't find the container with id c961bb28f855b20227752549aa28bc9152e5fad986d61dc1f83bf8a72af9be9c
Sep 09 16:09:22.890798 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:22.890773    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"]
Sep 09 16:09:22.897530 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:09:22.897505    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf547411_e2ec_45ab_b55b_8d8930e411e9.slice/crio-edc090bb58a3a882a4a9c1995b8b99e16fb60efdd30c91260421810b2059418d WatchSource:0}: Error finding container edc090bb58a3a882a4a9c1995b8b99e16fb60efdd30c91260421810b2059418d: Status 404 returned error can't find the container with id edc090bb58a3a882a4a9c1995b8b99e16fb60efdd30c91260421810b2059418d
Sep 09 16:09:23.460625 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:23.460589    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:23.652041 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:23.652005    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm" event={"ID":"bf547411-e2ec-45ab-b55b-8d8930e411e9","Type":"ContainerStarted","Data":"8667d0ec9eb229092437c49c18bd8019ab3fc42d46a3647dc46472d6f7179b58"}
Sep 09 16:09:23.652216 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:23.652069    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm" event={"ID":"bf547411-e2ec-45ab-b55b-8d8930e411e9","Type":"ContainerStarted","Data":"edc090bb58a3a882a4a9c1995b8b99e16fb60efdd30c91260421810b2059418d"}
Sep 09 16:09:23.652216 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:23.652101    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"
Sep 09 16:09:23.653591 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:23.653566    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt" event={"ID":"426dd61b-5ccc-4666-954c-05a6924e5524","Type":"ContainerStarted","Data":"3a257a2af1d3257e9f3940df9dffa0e4b445fb0ccd2c09148f17e4c20071af5e"}
Sep 09 16:09:23.653715 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:23.653617    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt" event={"ID":"426dd61b-5ccc-4666-954c-05a6924e5524","Type":"ContainerStarted","Data":"c961bb28f855b20227752549aa28bc9152e5fad986d61dc1f83bf8a72af9be9c"}
Sep 09 16:09:23.653830 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:23.653813    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"
Sep 09 16:09:23.669986 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:23.669918    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm" podStartSLOduration=1.669903486 podStartE2EDuration="1.669903486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:23.667883228 +0000 UTC m=+1068.534394448" watchObservedRunningTime="2026-09-09 16:09:23.669903486 +0000 UTC m=+1068.536414705"
Sep 09 16:09:23.685107 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:23.685061    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt" podStartSLOduration=1.6850476699999999 podStartE2EDuration="1.68504767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:23.684122235 +0000 UTC m=+1068.550633457" watchObservedRunningTime="2026-09-09 16:09:23.68504767 +0000 UTC m=+1068.551558887"
Sep 09 16:09:24.598483 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:24.598431    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"
Sep 09 16:09:24.850282 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:24.850192    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:24.949261 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:24.949218    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:25.171639 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:25.171551    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:26.644213 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:26.644177    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:26.786786 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:26.786746    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:26.885450 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:26.885415    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:29.137343 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:29.137250    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:29.616019 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:29.615967    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:29.838366 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:29.838335    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:31.083121 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:31.083081    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:31.148914 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:31.148881    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:31.236384 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:31.236350    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:31.377700 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:31.377614    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:31.508337 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:31.508290    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:31.609352 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:31.609305    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:33.273908 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:33.273872    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:33.655955 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:33.655870    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"
Sep 09 16:09:33.656082 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:33.656020    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"
Sep 09 16:09:34.662313 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:34.662277    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"
Sep 09 16:09:35.666266 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:35.666243    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"
Sep 09 16:09:35.897042 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:35.897002    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:36.253656 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:36.253624    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:36.584133 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:36.584053    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:37.032994 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:37.032955    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:37.129499 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:37.129429    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:37.741767 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:37.741734    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:37.842939 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:37.842901    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:40.003835 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:40.003797    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:40.103540 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:40.103499    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:41.159285 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:41.159250    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:41.262208 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:41.262169    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:41.495692 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:41.495659    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:41.765132 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:41.765059    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:41.866814 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:41.866775    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:41.878947 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:41.878915    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"]
Sep 09 16:09:41.883506 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:41.883488    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"
Sep 09 16:09:41.885924 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:41.885905    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c23ff1-dockercfg-tqnt7\""
Sep 09 16:09:41.886190 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:41.886174    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c23ff1\""
Sep 09 16:09:41.899882 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:41.899858    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"]
Sep 09 16:09:41.918171 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:41.918147    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"]
Sep 09 16:09:41.921490 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:41.921459    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"
Sep 09 16:09:41.932610 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:41.932590    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"]
Sep 09 16:09:41.958019 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:41.957990    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8677\" (UniqueName: \"kubernetes.io/projected/3ce4a718-2070-4af5-be6f-a498a7a38af5-kube-api-access-z8677\") pod \"payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj\" (UID: \"3ce4a718-2070-4af5-be6f-a498a7a38af5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"
Sep 09 16:09:41.958147 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:41.958029    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce4a718-2070-4af5-be6f-a498a7a38af5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj\" (UID: \"3ce4a718-2070-4af5-be6f-a498a7a38af5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"
Sep 09 16:09:42.059353 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.059252    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06afc164-5e10-4dba-b5fe-bc3d6fc921d5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48\" (UID: \"06afc164-5e10-4dba-b5fe-bc3d6fc921d5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"
Sep 09 16:09:42.059353 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.059306    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxlm\" (UniqueName: \"kubernetes.io/projected/06afc164-5e10-4dba-b5fe-bc3d6fc921d5-kube-api-access-tlxlm\") pod \"payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48\" (UID: \"06afc164-5e10-4dba-b5fe-bc3d6fc921d5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"
Sep 09 16:09:42.059582 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.059392    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8677\" (UniqueName: \"kubernetes.io/projected/3ce4a718-2070-4af5-be6f-a498a7a38af5-kube-api-access-z8677\") pod \"payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj\" (UID: \"3ce4a718-2070-4af5-be6f-a498a7a38af5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"
Sep 09 16:09:42.059582 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.059428    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce4a718-2070-4af5-be6f-a498a7a38af5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj\" (UID: \"3ce4a718-2070-4af5-be6f-a498a7a38af5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"
Sep 09 16:09:42.060130 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.060111    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce4a718-2070-4af5-be6f-a498a7a38af5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj\" (UID: \"3ce4a718-2070-4af5-be6f-a498a7a38af5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"
Sep 09 16:09:42.067748 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.067718    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8677\" (UniqueName: \"kubernetes.io/projected/3ce4a718-2070-4af5-be6f-a498a7a38af5-kube-api-access-z8677\") pod \"payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj\" (UID: \"3ce4a718-2070-4af5-be6f-a498a7a38af5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"
Sep 09 16:09:42.160853 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.160811    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06afc164-5e10-4dba-b5fe-bc3d6fc921d5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48\" (UID: \"06afc164-5e10-4dba-b5fe-bc3d6fc921d5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"
Sep 09 16:09:42.161263 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.160865    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxlm\" (UniqueName: \"kubernetes.io/projected/06afc164-5e10-4dba-b5fe-bc3d6fc921d5-kube-api-access-tlxlm\") pod \"payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48\" (UID: \"06afc164-5e10-4dba-b5fe-bc3d6fc921d5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"
Sep 09 16:09:42.161688 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.161665    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06afc164-5e10-4dba-b5fe-bc3d6fc921d5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48\" (UID: \"06afc164-5e10-4dba-b5fe-bc3d6fc921d5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"
Sep 09 16:09:42.176025 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.175999    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxlm\" (UniqueName: \"kubernetes.io/projected/06afc164-5e10-4dba-b5fe-bc3d6fc921d5-kube-api-access-tlxlm\") pod \"payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48\" (UID: \"06afc164-5e10-4dba-b5fe-bc3d6fc921d5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"
Sep 09 16:09:42.193868 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.193841    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"
Sep 09 16:09:42.231711 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.231680    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"
Sep 09 16:09:42.282866 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.282784    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:42.464825 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.464746    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"]
Sep 09 16:09:42.466813 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:09:42.466789    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce4a718_2070_4af5_be6f_a498a7a38af5.slice/crio-191aa8d900a2b2efdc0e5b9e48b243622597c373c23ff1520f315a8fa219b92d WatchSource:0}: Error finding container 191aa8d900a2b2efdc0e5b9e48b243622597c373c23ff1520f315a8fa219b92d: Status 404 returned error can't find the container with id 191aa8d900a2b2efdc0e5b9e48b243622597c373c23ff1520f315a8fa219b92d
Sep 09 16:09:42.534737 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.534677    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"]
Sep 09 16:09:42.536814 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:09:42.536788    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06afc164_5e10_4dba_b5fe_bc3d6fc921d5.slice/crio-aeb0935a48072f8944aa65c703c67af8fa15aa18219f22c7c678a107116e02ad WatchSource:0}: Error finding container aeb0935a48072f8944aa65c703c67af8fa15aa18219f22c7c678a107116e02ad: Status 404 returned error can't find the container with id aeb0935a48072f8944aa65c703c67af8fa15aa18219f22c7c678a107116e02ad
Sep 09 16:09:42.746594 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.746499    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48" event={"ID":"06afc164-5e10-4dba-b5fe-bc3d6fc921d5","Type":"ContainerStarted","Data":"6e40ee000a89000ccda417d53f75aa0deab56336a3c5a30c6711896760abcdaf"}
Sep 09 16:09:42.746594 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.746565    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48" event={"ID":"06afc164-5e10-4dba-b5fe-bc3d6fc921d5","Type":"ContainerStarted","Data":"aeb0935a48072f8944aa65c703c67af8fa15aa18219f22c7c678a107116e02ad"}
Sep 09 16:09:42.746594 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.746596    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"
Sep 09 16:09:42.748303 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.748271    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj" event={"ID":"3ce4a718-2070-4af5-be6f-a498a7a38af5","Type":"ContainerStarted","Data":"e3b854eca4ae49ebbbd1ffa73e3183f7643d9665cf7ac2c4ee41f0fd25e91a7b"}
Sep 09 16:09:42.748488 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.748328    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj" event={"ID":"3ce4a718-2070-4af5-be6f-a498a7a38af5","Type":"ContainerStarted","Data":"191aa8d900a2b2efdc0e5b9e48b243622597c373c23ff1520f315a8fa219b92d"}
Sep 09 16:09:42.748488 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.748389    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"
Sep 09 16:09:42.763193 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.763148    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48" podStartSLOduration=1.7631342239999999 podStartE2EDuration="1.763134224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:42.76150658 +0000 UTC m=+1087.628017797" watchObservedRunningTime="2026-09-09 16:09:42.763134224 +0000 UTC m=+1087.629645443"
Sep 09 16:09:42.779429 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:42.779381    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj" podStartSLOduration=1.779372079 podStartE2EDuration="1.779372079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:42.778548019 +0000 UTC m=+1087.645059258" watchObservedRunningTime="2026-09-09 16:09:42.779372079 +0000 UTC m=+1087.645883331"
Sep 09 16:09:43.092783 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:43.092679    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:43.873753 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:43.873718    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:45.156995 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:45.156959    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:45.208551 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:45.208516    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:45.451670 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:45.451634    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:46.167827 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:46.167783    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:50.027768 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:50.027731    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:09:53.754124 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:53.754091    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"
Sep 09 16:09:53.754516 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:09:53.754177    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"
Sep 09 16:10:00.996106 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:00.996068    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:01.013571 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.013535    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"]
Sep 09 16:10:01.013817 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.013791    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq" podUID="68e29f7c-6c42-4630-90f7-ef961ba55b5c" containerName="payload-processing" containerID="cri-o://184a50f55f64f93178e024fa1ff1a7e065f084a572fc41bdb7e7ee46a3127489" gracePeriod=30
Sep 09 16:10:01.050665 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.050628    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"]
Sep 09 16:10:01.050926 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.050882    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427" podUID="518bb67d-c896-4cb7-bebb-8bddd100b7df" containerName="payload-pre-processing" containerID="cri-o://00c7289ca2c74707058cc241a52ed50e9ac7c1406dbfbfeb01ded18892b66250" gracePeriod=30
Sep 09 16:10:01.074574 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.074542    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:01.133170 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.133136    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:01.194780 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.194720    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:01.306577 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.306527    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"
Sep 09 16:10:01.331836 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.331812    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e29f7c-6c42-4630-90f7-ef961ba55b5c-plugins-config-volume\") pod \"68e29f7c-6c42-4630-90f7-ef961ba55b5c\" (UID: \"68e29f7c-6c42-4630-90f7-ef961ba55b5c\") "
Sep 09 16:10:01.331984 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.331895    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cn2b9\" (UniqueName: \"kubernetes.io/projected/68e29f7c-6c42-4630-90f7-ef961ba55b5c-kube-api-access-cn2b9\") pod \"68e29f7c-6c42-4630-90f7-ef961ba55b5c\" (UID: \"68e29f7c-6c42-4630-90f7-ef961ba55b5c\") "
Sep 09 16:10:01.332188 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.332165    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e29f7c-6c42-4630-90f7-ef961ba55b5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68e29f7c-6c42-4630-90f7-ef961ba55b5c" (UID: "68e29f7c-6c42-4630-90f7-ef961ba55b5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:10:01.334451 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.334417    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e29f7c-6c42-4630-90f7-ef961ba55b5c-kube-api-access-cn2b9" (OuterVolumeSpecName: "kube-api-access-cn2b9") pod "68e29f7c-6c42-4630-90f7-ef961ba55b5c" (UID: "68e29f7c-6c42-4630-90f7-ef961ba55b5c"). InnerVolumeSpecName "kube-api-access-cn2b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:10:01.337942 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.337920    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"
Sep 09 16:10:01.375207 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.375167    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:01.432374 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.432328    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/518bb67d-c896-4cb7-bebb-8bddd100b7df-plugins-config-volume\") pod \"518bb67d-c896-4cb7-bebb-8bddd100b7df\" (UID: \"518bb67d-c896-4cb7-bebb-8bddd100b7df\") "
Sep 09 16:10:01.432374 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.432378    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t6r9\" (UniqueName: \"kubernetes.io/projected/518bb67d-c896-4cb7-bebb-8bddd100b7df-kube-api-access-4t6r9\") pod \"518bb67d-c896-4cb7-bebb-8bddd100b7df\" (UID: \"518bb67d-c896-4cb7-bebb-8bddd100b7df\") "
Sep 09 16:10:01.432674 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.432631    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn2b9\" (UniqueName: \"kubernetes.io/projected/68e29f7c-6c42-4630-90f7-ef961ba55b5c-kube-api-access-cn2b9\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:10:01.432674 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.432652    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e29f7c-6c42-4630-90f7-ef961ba55b5c-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:10:01.432766 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.432740    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/518bb67d-c896-4cb7-bebb-8bddd100b7df-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "518bb67d-c896-4cb7-bebb-8bddd100b7df" (UID: "518bb67d-c896-4cb7-bebb-8bddd100b7df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:10:01.434472 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.434438    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518bb67d-c896-4cb7-bebb-8bddd100b7df-kube-api-access-4t6r9" (OuterVolumeSpecName: "kube-api-access-4t6r9") pod "518bb67d-c896-4cb7-bebb-8bddd100b7df" (UID: "518bb67d-c896-4cb7-bebb-8bddd100b7df"). InnerVolumeSpecName "kube-api-access-4t6r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:10:01.533961 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.533879    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/518bb67d-c896-4cb7-bebb-8bddd100b7df-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:10:01.533961 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.533908    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t6r9\" (UniqueName: \"kubernetes.io/projected/518bb67d-c896-4cb7-bebb-8bddd100b7df-kube-api-access-4t6r9\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:10:01.793383 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.793292    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:01.835765 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.835731    2556 generic.go:356] "Generic (PLEG): container finished" podID="518bb67d-c896-4cb7-bebb-8bddd100b7df" containerID="00c7289ca2c74707058cc241a52ed50e9ac7c1406dbfbfeb01ded18892b66250" exitCode=0
Sep 09 16:10:01.835942 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.835798    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"
Sep 09 16:10:01.835942 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.835805    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427" event={"ID":"518bb67d-c896-4cb7-bebb-8bddd100b7df","Type":"ContainerDied","Data":"00c7289ca2c74707058cc241a52ed50e9ac7c1406dbfbfeb01ded18892b66250"}
Sep 09 16:10:01.835942 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.835858    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427" event={"ID":"518bb67d-c896-4cb7-bebb-8bddd100b7df","Type":"ContainerDied","Data":"425e3d6005358761adbcceecc35f842bf14670ef86958e5f527d3966b3dd7a1e"}
Sep 09 16:10:01.835942 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.835890    2556 scope.go:117] "RemoveContainer" containerID="00c7289ca2c74707058cc241a52ed50e9ac7c1406dbfbfeb01ded18892b66250"
Sep 09 16:10:01.837180 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.837154    2556 generic.go:356] "Generic (PLEG): container finished" podID="68e29f7c-6c42-4630-90f7-ef961ba55b5c" containerID="184a50f55f64f93178e024fa1ff1a7e065f084a572fc41bdb7e7ee46a3127489" exitCode=0
Sep 09 16:10:01.837307 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.837182    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq" event={"ID":"68e29f7c-6c42-4630-90f7-ef961ba55b5c","Type":"ContainerDied","Data":"184a50f55f64f93178e024fa1ff1a7e065f084a572fc41bdb7e7ee46a3127489"}
Sep 09 16:10:01.837307 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.837219    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"
Sep 09 16:10:01.837307 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.837225    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq" event={"ID":"68e29f7c-6c42-4630-90f7-ef961ba55b5c","Type":"ContainerDied","Data":"c29026f2d897ee3c75ecfdd023ba737e54df6e389c311aa20769b7b2549255f0"}
Sep 09 16:10:01.844941 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.844924    2556 scope.go:117] "RemoveContainer" containerID="00c7289ca2c74707058cc241a52ed50e9ac7c1406dbfbfeb01ded18892b66250"
Sep 09 16:10:01.845236 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:10:01.845216    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c7289ca2c74707058cc241a52ed50e9ac7c1406dbfbfeb01ded18892b66250\": container with ID starting with 00c7289ca2c74707058cc241a52ed50e9ac7c1406dbfbfeb01ded18892b66250 not found: ID does not exist" containerID="00c7289ca2c74707058cc241a52ed50e9ac7c1406dbfbfeb01ded18892b66250"
Sep 09 16:10:01.845304 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.845245    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c7289ca2c74707058cc241a52ed50e9ac7c1406dbfbfeb01ded18892b66250"} err="failed to get container status \"00c7289ca2c74707058cc241a52ed50e9ac7c1406dbfbfeb01ded18892b66250\": rpc error: code = NotFound desc = could not find container \"00c7289ca2c74707058cc241a52ed50e9ac7c1406dbfbfeb01ded18892b66250\": container with ID starting with 00c7289ca2c74707058cc241a52ed50e9ac7c1406dbfbfeb01ded18892b66250 not found: ID does not exist"
Sep 09 16:10:01.845304 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.845264    2556 scope.go:117] "RemoveContainer" containerID="184a50f55f64f93178e024fa1ff1a7e065f084a572fc41bdb7e7ee46a3127489"
Sep 09 16:10:01.853199 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.853178    2556 scope.go:117] "RemoveContainer" containerID="184a50f55f64f93178e024fa1ff1a7e065f084a572fc41bdb7e7ee46a3127489"
Sep 09 16:10:01.853518 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:10:01.853497    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184a50f55f64f93178e024fa1ff1a7e065f084a572fc41bdb7e7ee46a3127489\": container with ID starting with 184a50f55f64f93178e024fa1ff1a7e065f084a572fc41bdb7e7ee46a3127489 not found: ID does not exist" containerID="184a50f55f64f93178e024fa1ff1a7e065f084a572fc41bdb7e7ee46a3127489"
Sep 09 16:10:01.853587 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.853528    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184a50f55f64f93178e024fa1ff1a7e065f084a572fc41bdb7e7ee46a3127489"} err="failed to get container status \"184a50f55f64f93178e024fa1ff1a7e065f084a572fc41bdb7e7ee46a3127489\": rpc error: code = NotFound desc = could not find container \"184a50f55f64f93178e024fa1ff1a7e065f084a572fc41bdb7e7ee46a3127489\": container with ID starting with 184a50f55f64f93178e024fa1ff1a7e065f084a572fc41bdb7e7ee46a3127489 not found: ID does not exist"
Sep 09 16:10:01.854134 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.854115    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"]
Sep 09 16:10:01.857563 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.857538    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-083463c6-f65b69cf4-sf427"]
Sep 09 16:10:01.866931 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.866906    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"]
Sep 09 16:10:01.870298 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:01.870276    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-083463c6-699755b9b5-nvcxq"]
Sep 09 16:10:02.172005 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:02.171925    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:02.340034 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:02.340001    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:02.496864 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:02.496823    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:02.643454 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:02.643421    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:03.763097 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:03.763065    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="518bb67d-c896-4cb7-bebb-8bddd100b7df" path="/var/lib/kubelet/pods/518bb67d-c896-4cb7-bebb-8bddd100b7df/volumes"
Sep 09 16:10:03.763495 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:03.763408    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68e29f7c-6c42-4630-90f7-ef961ba55b5c" path="/var/lib/kubelet/pods/68e29f7c-6c42-4630-90f7-ef961ba55b5c/volumes"
Sep 09 16:10:04.707871 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:04.707835    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:04.810914 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:04.810881    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:04.908580 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:04.908536    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:05.008535 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.008432    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:05.109347 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.109297    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:05.469852 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.469815    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:05.562668 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.562637    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:05.562889 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.562870    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"]
Sep 09 16:10:05.563247 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.563234    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e29f7c-6c42-4630-90f7-ef961ba55b5c" containerName="payload-processing"
Sep 09 16:10:05.563284 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.563251    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e29f7c-6c42-4630-90f7-ef961ba55b5c" containerName="payload-processing"
Sep 09 16:10:05.563284 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.563268    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="518bb67d-c896-4cb7-bebb-8bddd100b7df" containerName="payload-pre-processing"
Sep 09 16:10:05.563284 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.563274    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="518bb67d-c896-4cb7-bebb-8bddd100b7df" containerName="payload-pre-processing"
Sep 09 16:10:05.563378 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.563341    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="68e29f7c-6c42-4630-90f7-ef961ba55b5c" containerName="payload-processing"
Sep 09 16:10:05.563378 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.563352    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="518bb67d-c896-4cb7-bebb-8bddd100b7df" containerName="payload-pre-processing"
Sep 09 16:10:05.566501 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.566458    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"
Sep 09 16:10:05.568723 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.568702    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-dae90c4d-dockercfg-9ngld\""
Sep 09 16:10:05.568836 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.568750    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-dae90c4d\""
Sep 09 16:10:05.578001 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.577973    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"]
Sep 09 16:10:05.586138 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.586108    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"]
Sep 09 16:10:05.589315 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.589293    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"
Sep 09 16:10:05.600642 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.600617    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"]
Sep 09 16:10:05.671521 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.671488    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6lkh\" (UniqueName: \"kubernetes.io/projected/ee25e0dd-69bf-4079-9d23-9c137c077b4e-kube-api-access-l6lkh\") pod \"payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t\" (UID: \"ee25e0dd-69bf-4079-9d23-9c137c077b4e\") " pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"
Sep 09 16:10:05.671687 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.671544    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv\" (UID: \"3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"
Sep 09 16:10:05.671687 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.671594    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee25e0dd-69bf-4079-9d23-9c137c077b4e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t\" (UID: \"ee25e0dd-69bf-4079-9d23-9c137c077b4e\") " pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"
Sep 09 16:10:05.671687 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.671613    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qgsm\" (UniqueName: \"kubernetes.io/projected/3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2-kube-api-access-9qgsm\") pod \"payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv\" (UID: \"3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"
Sep 09 16:10:05.772760 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.772675    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee25e0dd-69bf-4079-9d23-9c137c077b4e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t\" (UID: \"ee25e0dd-69bf-4079-9d23-9c137c077b4e\") " pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"
Sep 09 16:10:05.772760 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.772714    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qgsm\" (UniqueName: \"kubernetes.io/projected/3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2-kube-api-access-9qgsm\") pod \"payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv\" (UID: \"3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"
Sep 09 16:10:05.772968 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.772769    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6lkh\" (UniqueName: \"kubernetes.io/projected/ee25e0dd-69bf-4079-9d23-9c137c077b4e-kube-api-access-l6lkh\") pod \"payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t\" (UID: \"ee25e0dd-69bf-4079-9d23-9c137c077b4e\") " pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"
Sep 09 16:10:05.772968 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.772799    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv\" (UID: \"3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"
Sep 09 16:10:05.773409 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.773390    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee25e0dd-69bf-4079-9d23-9c137c077b4e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t\" (UID: \"ee25e0dd-69bf-4079-9d23-9c137c077b4e\") " pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"
Sep 09 16:10:05.773499 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.773480    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv\" (UID: \"3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"
Sep 09 16:10:05.780770 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.780736    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6lkh\" (UniqueName: \"kubernetes.io/projected/ee25e0dd-69bf-4079-9d23-9c137c077b4e-kube-api-access-l6lkh\") pod \"payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t\" (UID: \"ee25e0dd-69bf-4079-9d23-9c137c077b4e\") " pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"
Sep 09 16:10:05.780994 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.780974    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qgsm\" (UniqueName: \"kubernetes.io/projected/3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2-kube-api-access-9qgsm\") pod \"payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv\" (UID: \"3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"
Sep 09 16:10:05.876616 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.876585    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"
Sep 09 16:10:05.900625 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:05.900595    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"
Sep 09 16:10:06.152277 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:06.152232    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"]
Sep 09 16:10:06.154552 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:10:06.154514    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3894ef04_7b5d_43eb_bd2b_fb37dafd8ab2.slice/crio-7853b4e59208f3fead73aa27156a94590b2c2fd36961a0dff432c7458cdf067d WatchSource:0}: Error finding container 7853b4e59208f3fead73aa27156a94590b2c2fd36961a0dff432c7458cdf067d: Status 404 returned error can't find the container with id 7853b4e59208f3fead73aa27156a94590b2c2fd36961a0dff432c7458cdf067d
Sep 09 16:10:06.179770 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:06.179747    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"]
Sep 09 16:10:06.182247 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:10:06.182218    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee25e0dd_69bf_4079_9d23_9c137c077b4e.slice/crio-dc856de6354c4f08dc4f4d01835e83eda628449c9d5a3905c146f05382798763 WatchSource:0}: Error finding container dc856de6354c4f08dc4f4d01835e83eda628449c9d5a3905c146f05382798763: Status 404 returned error can't find the container with id dc856de6354c4f08dc4f4d01835e83eda628449c9d5a3905c146f05382798763
Sep 09 16:10:06.863924 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:06.863886    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t" event={"ID":"ee25e0dd-69bf-4079-9d23-9c137c077b4e","Type":"ContainerStarted","Data":"da8fb388d4e3ce4827dcb1b93c3584b1bab5ed98661a6fafa4c40946459a92be"}
Sep 09 16:10:06.864136 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:06.863940    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t" event={"ID":"ee25e0dd-69bf-4079-9d23-9c137c077b4e","Type":"ContainerStarted","Data":"dc856de6354c4f08dc4f4d01835e83eda628449c9d5a3905c146f05382798763"}
Sep 09 16:10:06.864136 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:06.864016    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"
Sep 09 16:10:06.865639 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:06.865613    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv" event={"ID":"3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2","Type":"ContainerStarted","Data":"c7d401fcc43e39bc796d5862ff970177fcf358048e9c159f0c8010fa4465fecf"}
Sep 09 16:10:06.865741 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:06.865667    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"
Sep 09 16:10:06.865741 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:06.865685    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv" event={"ID":"3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2","Type":"ContainerStarted","Data":"7853b4e59208f3fead73aa27156a94590b2c2fd36961a0dff432c7458cdf067d"}
Sep 09 16:10:06.884421 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:06.884370    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t" podStartSLOduration=1.884355693 podStartE2EDuration="1.884355693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:10:06.883242617 +0000 UTC m=+1111.749753840" watchObservedRunningTime="2026-09-09 16:10:06.884355693 +0000 UTC m=+1111.750866913"
Sep 09 16:10:06.901154 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:06.901080    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv" podStartSLOduration=1.901061743 podStartE2EDuration="1.901061743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:10:06.8982287 +0000 UTC m=+1111.764739921" watchObservedRunningTime="2026-09-09 16:10:06.901061743 +0000 UTC m=+1111.767572966"
Sep 09 16:10:17.872746 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:17.872714    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"
Sep 09 16:10:17.873338 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:17.872831    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"
Sep 09 16:10:36.282913 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.282870    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:36.290123 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.290091    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"]
Sep 09 16:10:36.290379 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.290330    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t" podUID="ee25e0dd-69bf-4079-9d23-9c137c077b4e" containerName="payload-processing" containerID="cri-o://da8fb388d4e3ce4827dcb1b93c3584b1bab5ed98661a6fafa4c40946459a92be" gracePeriod=30
Sep 09 16:10:36.298918 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.298892    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"]
Sep 09 16:10:36.299283 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.299257    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv" podUID="3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2" containerName="payload-pre-processing" containerID="cri-o://c7d401fcc43e39bc796d5862ff970177fcf358048e9c159f0c8010fa4465fecf" gracePeriod=30
Sep 09 16:10:36.382652 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.382607    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:36.452397 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.452357    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:36.538983 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.538924    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"
Sep 09 16:10:36.575016 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.574984    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"
Sep 09 16:10:36.575951 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.575924    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:36.634949 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.634916    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6lkh\" (UniqueName: \"kubernetes.io/projected/ee25e0dd-69bf-4079-9d23-9c137c077b4e-kube-api-access-l6lkh\") pod \"ee25e0dd-69bf-4079-9d23-9c137c077b4e\" (UID: \"ee25e0dd-69bf-4079-9d23-9c137c077b4e\") "
Sep 09 16:10:36.635118 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.634983    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee25e0dd-69bf-4079-9d23-9c137c077b4e-plugins-config-volume\") pod \"ee25e0dd-69bf-4079-9d23-9c137c077b4e\" (UID: \"ee25e0dd-69bf-4079-9d23-9c137c077b4e\") "
Sep 09 16:10:36.635390 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.635357    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee25e0dd-69bf-4079-9d23-9c137c077b4e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee25e0dd-69bf-4079-9d23-9c137c077b4e" (UID: "ee25e0dd-69bf-4079-9d23-9c137c077b4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:10:36.637012 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.636993    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee25e0dd-69bf-4079-9d23-9c137c077b4e-kube-api-access-l6lkh" (OuterVolumeSpecName: "kube-api-access-l6lkh") pod "ee25e0dd-69bf-4079-9d23-9c137c077b4e" (UID: "ee25e0dd-69bf-4079-9d23-9c137c077b4e"). InnerVolumeSpecName "kube-api-access-l6lkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:10:36.673971 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.673938    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:36.736332 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.736289    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2-plugins-config-volume\") pod \"3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2\" (UID: \"3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2\") "
Sep 09 16:10:36.736548 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.736343    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qgsm\" (UniqueName: \"kubernetes.io/projected/3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2-kube-api-access-9qgsm\") pod \"3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2\" (UID: \"3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2\") "
Sep 09 16:10:36.736620 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.736546    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee25e0dd-69bf-4079-9d23-9c137c077b4e-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:10:36.736620 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.736564    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6lkh\" (UniqueName: \"kubernetes.io/projected/ee25e0dd-69bf-4079-9d23-9c137c077b4e-kube-api-access-l6lkh\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:10:36.736701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.736662    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2" (UID: "3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:10:36.738347 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.738326    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2-kube-api-access-9qgsm" (OuterVolumeSpecName: "kube-api-access-9qgsm") pod "3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2" (UID: "3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2"). InnerVolumeSpecName "kube-api-access-9qgsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:10:36.837151 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.837067    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:10:36.837151 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:36.837096    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qgsm\" (UniqueName: \"kubernetes.io/projected/3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2-kube-api-access-9qgsm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:10:37.002757 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.002725    2556 generic.go:356] "Generic (PLEG): container finished" podID="ee25e0dd-69bf-4079-9d23-9c137c077b4e" containerID="da8fb388d4e3ce4827dcb1b93c3584b1bab5ed98661a6fafa4c40946459a92be" exitCode=0
Sep 09 16:10:37.002945 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.002771    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"
Sep 09 16:10:37.002945 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.002814    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t" event={"ID":"ee25e0dd-69bf-4079-9d23-9c137c077b4e","Type":"ContainerDied","Data":"da8fb388d4e3ce4827dcb1b93c3584b1bab5ed98661a6fafa4c40946459a92be"}
Sep 09 16:10:37.002945 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.002868    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t" event={"ID":"ee25e0dd-69bf-4079-9d23-9c137c077b4e","Type":"ContainerDied","Data":"dc856de6354c4f08dc4f4d01835e83eda628449c9d5a3905c146f05382798763"}
Sep 09 16:10:37.002945 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.002895    2556 scope.go:117] "RemoveContainer" containerID="da8fb388d4e3ce4827dcb1b93c3584b1bab5ed98661a6fafa4c40946459a92be"
Sep 09 16:10:37.004211 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.004106    2556 generic.go:356] "Generic (PLEG): container finished" podID="3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2" containerID="c7d401fcc43e39bc796d5862ff970177fcf358048e9c159f0c8010fa4465fecf" exitCode=0
Sep 09 16:10:37.004211 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.004149    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv" event={"ID":"3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2","Type":"ContainerDied","Data":"c7d401fcc43e39bc796d5862ff970177fcf358048e9c159f0c8010fa4465fecf"}
Sep 09 16:10:37.004211 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.004179    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv" event={"ID":"3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2","Type":"ContainerDied","Data":"7853b4e59208f3fead73aa27156a94590b2c2fd36961a0dff432c7458cdf067d"}
Sep 09 16:10:37.004369 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.004219    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"
Sep 09 16:10:37.011431 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.011411    2556 scope.go:117] "RemoveContainer" containerID="da8fb388d4e3ce4827dcb1b93c3584b1bab5ed98661a6fafa4c40946459a92be"
Sep 09 16:10:37.011770 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:10:37.011748    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da8fb388d4e3ce4827dcb1b93c3584b1bab5ed98661a6fafa4c40946459a92be\": container with ID starting with da8fb388d4e3ce4827dcb1b93c3584b1bab5ed98661a6fafa4c40946459a92be not found: ID does not exist" containerID="da8fb388d4e3ce4827dcb1b93c3584b1bab5ed98661a6fafa4c40946459a92be"
Sep 09 16:10:37.011877 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.011776    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8fb388d4e3ce4827dcb1b93c3584b1bab5ed98661a6fafa4c40946459a92be"} err="failed to get container status \"da8fb388d4e3ce4827dcb1b93c3584b1bab5ed98661a6fafa4c40946459a92be\": rpc error: code = NotFound desc = could not find container \"da8fb388d4e3ce4827dcb1b93c3584b1bab5ed98661a6fafa4c40946459a92be\": container with ID starting with da8fb388d4e3ce4827dcb1b93c3584b1bab5ed98661a6fafa4c40946459a92be not found: ID does not exist"
Sep 09 16:10:37.011877 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.011791    2556 scope.go:117] "RemoveContainer" containerID="c7d401fcc43e39bc796d5862ff970177fcf358048e9c159f0c8010fa4465fecf"
Sep 09 16:10:37.022063 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.022046    2556 scope.go:117] "RemoveContainer" containerID="c7d401fcc43e39bc796d5862ff970177fcf358048e9c159f0c8010fa4465fecf"
Sep 09 16:10:37.022358 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:10:37.022335    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d401fcc43e39bc796d5862ff970177fcf358048e9c159f0c8010fa4465fecf\": container with ID starting with c7d401fcc43e39bc796d5862ff970177fcf358048e9c159f0c8010fa4465fecf not found: ID does not exist" containerID="c7d401fcc43e39bc796d5862ff970177fcf358048e9c159f0c8010fa4465fecf"
Sep 09 16:10:37.022448 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.022362    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d401fcc43e39bc796d5862ff970177fcf358048e9c159f0c8010fa4465fecf"} err="failed to get container status \"c7d401fcc43e39bc796d5862ff970177fcf358048e9c159f0c8010fa4465fecf\": rpc error: code = NotFound desc = could not find container \"c7d401fcc43e39bc796d5862ff970177fcf358048e9c159f0c8010fa4465fecf\": container with ID starting with c7d401fcc43e39bc796d5862ff970177fcf358048e9c159f0c8010fa4465fecf not found: ID does not exist"
Sep 09 16:10:37.034524 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.034492    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"]
Sep 09 16:10:37.041479 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.041436    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dae90c4d-6869947cbb-dts5t"]
Sep 09 16:10:37.070387 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.070352    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"]
Sep 09 16:10:37.078619 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.078588    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dae90c4d-69ffddfcfd-6gfxv"]
Sep 09 16:10:37.113324 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.113249    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:37.763957 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.763923    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2" path="/var/lib/kubelet/pods/3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2/volumes"
Sep 09 16:10:37.764352 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:37.764268    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee25e0dd-69bf-4079-9d23-9c137c077b4e" path="/var/lib/kubelet/pods/ee25e0dd-69bf-4079-9d23-9c137c077b4e/volumes"
Sep 09 16:10:38.322671 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:38.322634    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:38.424300 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:38.424267    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:39.906277 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:39.906237    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:40.004285 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:40.004246    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:41.070960 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.070916    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:41.172501 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.172457    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:41.425728 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.425698    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:41.652647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.652606    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"]
Sep 09 16:10:41.652951 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.652939    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee25e0dd-69bf-4079-9d23-9c137c077b4e" containerName="payload-processing"
Sep 09 16:10:41.653008 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.652952    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee25e0dd-69bf-4079-9d23-9c137c077b4e" containerName="payload-processing"
Sep 09 16:10:41.653008 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.652970    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2" containerName="payload-pre-processing"
Sep 09 16:10:41.653008 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.652977    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2" containerName="payload-pre-processing"
Sep 09 16:10:41.653097 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.653030    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3894ef04-7b5d-43eb-bd2b-fb37dafd8ab2" containerName="payload-pre-processing"
Sep 09 16:10:41.653097 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.653037    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee25e0dd-69bf-4079-9d23-9c137c077b4e" containerName="payload-processing"
Sep 09 16:10:41.657256 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.657236    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"
Sep 09 16:10:41.660175 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.660156    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-58050ce1\""
Sep 09 16:10:41.660428 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.660409    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-58050ce1-dockercfg-gt677\""
Sep 09 16:10:41.669534 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.669508    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"]
Sep 09 16:10:41.706000 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.705935    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"]
Sep 09 16:10:41.709252 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.709235    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"
Sep 09 16:10:41.723390 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.723368    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"]
Sep 09 16:10:41.784281 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.784251    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfkx9\" (UniqueName: \"kubernetes.io/projected/25aa6957-fb2d-40ef-8784-ae0e9ab28760-kube-api-access-bfkx9\") pod \"payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7\" (UID: \"25aa6957-fb2d-40ef-8784-ae0e9ab28760\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"
Sep 09 16:10:41.784475 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.784359    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25aa6957-fb2d-40ef-8784-ae0e9ab28760-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7\" (UID: \"25aa6957-fb2d-40ef-8784-ae0e9ab28760\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"
Sep 09 16:10:41.884886 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.884842    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkx9\" (UniqueName: \"kubernetes.io/projected/25aa6957-fb2d-40ef-8784-ae0e9ab28760-kube-api-access-bfkx9\") pod \"payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7\" (UID: \"25aa6957-fb2d-40ef-8784-ae0e9ab28760\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"
Sep 09 16:10:41.885084 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.884930    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwq8d\" (UniqueName: \"kubernetes.io/projected/ee3f87fc-8a34-4fa7-a23b-97268adcc33a-kube-api-access-vwq8d\") pod \"payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn\" (UID: \"ee3f87fc-8a34-4fa7-a23b-97268adcc33a\") " pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"
Sep 09 16:10:41.885084 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.884990    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3f87fc-8a34-4fa7-a23b-97268adcc33a-plugins-config-volume\") pod \"payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn\" (UID: \"ee3f87fc-8a34-4fa7-a23b-97268adcc33a\") " pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"
Sep 09 16:10:41.885084 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.885054    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25aa6957-fb2d-40ef-8784-ae0e9ab28760-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7\" (UID: \"25aa6957-fb2d-40ef-8784-ae0e9ab28760\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"
Sep 09 16:10:41.885737 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.885719    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25aa6957-fb2d-40ef-8784-ae0e9ab28760-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7\" (UID: \"25aa6957-fb2d-40ef-8784-ae0e9ab28760\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"
Sep 09 16:10:41.893880 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.893843    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkx9\" (UniqueName: \"kubernetes.io/projected/25aa6957-fb2d-40ef-8784-ae0e9ab28760-kube-api-access-bfkx9\") pod \"payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7\" (UID: \"25aa6957-fb2d-40ef-8784-ae0e9ab28760\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"
Sep 09 16:10:41.967663 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.967562    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"
Sep 09 16:10:41.986305 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.986275    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwq8d\" (UniqueName: \"kubernetes.io/projected/ee3f87fc-8a34-4fa7-a23b-97268adcc33a-kube-api-access-vwq8d\") pod \"payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn\" (UID: \"ee3f87fc-8a34-4fa7-a23b-97268adcc33a\") " pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"
Sep 09 16:10:41.986433 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.986318    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3f87fc-8a34-4fa7-a23b-97268adcc33a-plugins-config-volume\") pod \"payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn\" (UID: \"ee3f87fc-8a34-4fa7-a23b-97268adcc33a\") " pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"
Sep 09 16:10:41.987047 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.987024    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3f87fc-8a34-4fa7-a23b-97268adcc33a-plugins-config-volume\") pod \"payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn\" (UID: \"ee3f87fc-8a34-4fa7-a23b-97268adcc33a\") " pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"
Sep 09 16:10:41.997055 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:41.997029    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwq8d\" (UniqueName: \"kubernetes.io/projected/ee3f87fc-8a34-4fa7-a23b-97268adcc33a-kube-api-access-vwq8d\") pod \"payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn\" (UID: \"ee3f87fc-8a34-4fa7-a23b-97268adcc33a\") " pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"
Sep 09 16:10:42.019306 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:42.019282    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"
Sep 09 16:10:42.238559 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:42.238533    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"]
Sep 09 16:10:42.240863 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:10:42.240831    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25aa6957_fb2d_40ef_8784_ae0e9ab28760.slice/crio-82dfc6720a925c21a17c5deda1ea87a0178f250c4e3d01924ad48e0fa63b1773 WatchSource:0}: Error finding container 82dfc6720a925c21a17c5deda1ea87a0178f250c4e3d01924ad48e0fa63b1773: Status 404 returned error can't find the container with id 82dfc6720a925c21a17c5deda1ea87a0178f250c4e3d01924ad48e0fa63b1773
Sep 09 16:10:42.302285 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:42.302260    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"]
Sep 09 16:10:42.305816 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:10:42.305791    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3f87fc_8a34_4fa7_a23b_97268adcc33a.slice/crio-942d6237b85c86c0536d94b095f92d0eba6961119507a27067e089eb4bf8c989 WatchSource:0}: Error finding container 942d6237b85c86c0536d94b095f92d0eba6961119507a27067e089eb4bf8c989: Status 404 returned error can't find the container with id 942d6237b85c86c0536d94b095f92d0eba6961119507a27067e089eb4bf8c989
Sep 09 16:10:43.041553 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:43.041458    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn" event={"ID":"ee3f87fc-8a34-4fa7-a23b-97268adcc33a","Type":"ContainerStarted","Data":"ffbded5fdb7978bd26e63ce6ee16349e029775d58d1c8a3957d10315eb4cc6b3"}
Sep 09 16:10:43.041553 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:43.041552    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"
Sep 09 16:10:43.041815 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:43.041575    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn" event={"ID":"ee3f87fc-8a34-4fa7-a23b-97268adcc33a","Type":"ContainerStarted","Data":"942d6237b85c86c0536d94b095f92d0eba6961119507a27067e089eb4bf8c989"}
Sep 09 16:10:43.043104 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:43.043075    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7" event={"ID":"25aa6957-fb2d-40ef-8784-ae0e9ab28760","Type":"ContainerStarted","Data":"c2b6b326f45473d857bd2f3e681a76a27c26ef14136e3496babc6602693d9ac0"}
Sep 09 16:10:43.043244 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:43.043115    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7" event={"ID":"25aa6957-fb2d-40ef-8784-ae0e9ab28760","Type":"ContainerStarted","Data":"82dfc6720a925c21a17c5deda1ea87a0178f250c4e3d01924ad48e0fa63b1773"}
Sep 09 16:10:43.043244 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:43.043190    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"
Sep 09 16:10:43.060173 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:43.060117    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn" podStartSLOduration=2.060100009 podStartE2EDuration="2.060100009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:10:43.058619129 +0000 UTC m=+1147.925130382" watchObservedRunningTime="2026-09-09 16:10:43.060100009 +0000 UTC m=+1147.926611229"
Sep 09 16:10:43.084406 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:43.084356    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7" podStartSLOduration=2.084340214 podStartE2EDuration="2.084340214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:10:43.082852539 +0000 UTC m=+1147.949363758" watchObservedRunningTime="2026-09-09 16:10:43.084340214 +0000 UTC m=+1147.950851433"
Sep 09 16:10:43.203134 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:43.203093    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:43.456559 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:43.456522    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:43.826038 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:43.825937    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"]
Sep 09 16:10:43.830064 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:43.830041    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:43.834046 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:43.834018    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-049e4f89-kserve-self-signed-certs\""
Sep 09 16:10:43.849888 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:43.849853    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"]
Sep 09 16:10:44.002936 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.002903    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkcqw\" (UniqueName: \"kubernetes.io/projected/3d73e719-8b8d-4a10-9431-13d272a667ab-kube-api-access-gkcqw\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.003147 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.002954    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-model-cache\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.003147 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.003043    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-home\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.003147 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.003068    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-tmp-dir\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.003147 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.003100    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-dshm\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.003147 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.003117    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d73e719-8b8d-4a10-9431-13d272a667ab-tls-certs\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.103701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.103606    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-model-cache\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.103701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.103692    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-home\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.103928 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.103716    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-tmp-dir\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.103928 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.103751    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-dshm\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.103928 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.103776    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d73e719-8b8d-4a10-9431-13d272a667ab-tls-certs\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.103928 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.103832    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkcqw\" (UniqueName: \"kubernetes.io/projected/3d73e719-8b8d-4a10-9431-13d272a667ab-kube-api-access-gkcqw\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.104143 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.104097    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-home\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.104143 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.104129    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-model-cache\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.104254 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.104155    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-tmp-dir\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.105951 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.105926    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-dshm\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.106305 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.106285    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d73e719-8b8d-4a10-9431-13d272a667ab-tls-certs\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.112621 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.112599    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkcqw\" (UniqueName: \"kubernetes.io/projected/3d73e719-8b8d-4a10-9431-13d272a667ab-kube-api-access-gkcqw\") pod \"e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") " pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.140643 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.140612    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:44.326273 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.326219    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:10:44.409881 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:44.409840    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"]
Sep 09 16:10:44.412605 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:10:44.412574    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d73e719_8b8d_4a10_9431_13d272a667ab.slice/crio-0facb051afa04be2422a15830bec30deb0d260dac54bebe6edeabfb0c53b3b30 WatchSource:0}: Error finding container 0facb051afa04be2422a15830bec30deb0d260dac54bebe6edeabfb0c53b3b30: Status 404 returned error can't find the container with id 0facb051afa04be2422a15830bec30deb0d260dac54bebe6edeabfb0c53b3b30
Sep 09 16:10:45.053865 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:45.053832    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v" event={"ID":"3d73e719-8b8d-4a10-9431-13d272a667ab","Type":"ContainerStarted","Data":"1da76417bde985ceb87bef12b46c1a338b78c41af88008248fe76dd80eade84a"}
Sep 09 16:10:45.054244 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:45.053874    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v" event={"ID":"3d73e719-8b8d-4a10-9431-13d272a667ab","Type":"ContainerStarted","Data":"0facb051afa04be2422a15830bec30deb0d260dac54bebe6edeabfb0c53b3b30"}
Sep 09 16:10:45.071773 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:45.071726    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v" podStartSLOduration=2.071711146 podStartE2EDuration="2.071711146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:10:45.070308119 +0000 UTC m=+1149.936819339" watchObservedRunningTime="2026-09-09 16:10:45.071711146 +0000 UTC m=+1149.938222367"
Sep 09 16:10:54.050439 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:54.050401    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"
Sep 09 16:10:54.050983 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:54.050587    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"
Sep 09 16:10:54.141149 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:54.141112    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:54.141336 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:54.141176    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:54.153855 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:54.153829    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:10:55.109510 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:10:55.109483    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:11:03.903021 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:03.902933    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:11.796089 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:11.796051    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:11.827133 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:11.827103    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"]
Sep 09 16:11:11.827366 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:11.827317    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn" podUID="ee3f87fc-8a34-4fa7-a23b-97268adcc33a" containerName="payload-processing" containerID="cri-o://ffbded5fdb7978bd26e63ce6ee16349e029775d58d1c8a3957d10315eb4cc6b3" gracePeriod=30
Sep 09 16:11:11.833085 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:11.833061    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"]
Sep 09 16:11:11.833298 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:11.833252    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7" podUID="25aa6957-fb2d-40ef-8784-ae0e9ab28760" containerName="payload-pre-processing" containerID="cri-o://c2b6b326f45473d857bd2f3e681a76a27c26ef14136e3496babc6602693d9ac0" gracePeriod=30
Sep 09 16:11:11.870937 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:11.870898    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:11.946808 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:11.946775    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:12.030683 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.030649    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:12.097494 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.097456    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"
Sep 09 16:11:12.100287 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.100269    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"
Sep 09 16:11:12.114002 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.113971    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25aa6957-fb2d-40ef-8784-ae0e9ab28760-plugins-config-volume\") pod \"25aa6957-fb2d-40ef-8784-ae0e9ab28760\" (UID: \"25aa6957-fb2d-40ef-8784-ae0e9ab28760\") "
Sep 09 16:11:12.114147 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.114006    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfkx9\" (UniqueName: \"kubernetes.io/projected/25aa6957-fb2d-40ef-8784-ae0e9ab28760-kube-api-access-bfkx9\") pod \"25aa6957-fb2d-40ef-8784-ae0e9ab28760\" (UID: \"25aa6957-fb2d-40ef-8784-ae0e9ab28760\") "
Sep 09 16:11:12.114147 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.114047    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwq8d\" (UniqueName: \"kubernetes.io/projected/ee3f87fc-8a34-4fa7-a23b-97268adcc33a-kube-api-access-vwq8d\") pod \"ee3f87fc-8a34-4fa7-a23b-97268adcc33a\" (UID: \"ee3f87fc-8a34-4fa7-a23b-97268adcc33a\") "
Sep 09 16:11:12.114147 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.114072    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3f87fc-8a34-4fa7-a23b-97268adcc33a-plugins-config-volume\") pod \"ee3f87fc-8a34-4fa7-a23b-97268adcc33a\" (UID: \"ee3f87fc-8a34-4fa7-a23b-97268adcc33a\") "
Sep 09 16:11:12.114436 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.114409    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25aa6957-fb2d-40ef-8784-ae0e9ab28760-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25aa6957-fb2d-40ef-8784-ae0e9ab28760" (UID: "25aa6957-fb2d-40ef-8784-ae0e9ab28760"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:11:12.114571 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.114443    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3f87fc-8a34-4fa7-a23b-97268adcc33a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee3f87fc-8a34-4fa7-a23b-97268adcc33a" (UID: "ee3f87fc-8a34-4fa7-a23b-97268adcc33a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:11:12.116586 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.116547    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25aa6957-fb2d-40ef-8784-ae0e9ab28760-kube-api-access-bfkx9" (OuterVolumeSpecName: "kube-api-access-bfkx9") pod "25aa6957-fb2d-40ef-8784-ae0e9ab28760" (UID: "25aa6957-fb2d-40ef-8784-ae0e9ab28760"). InnerVolumeSpecName "kube-api-access-bfkx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:11:12.116743 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.116712    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3f87fc-8a34-4fa7-a23b-97268adcc33a-kube-api-access-vwq8d" (OuterVolumeSpecName: "kube-api-access-vwq8d") pod "ee3f87fc-8a34-4fa7-a23b-97268adcc33a" (UID: "ee3f87fc-8a34-4fa7-a23b-97268adcc33a"). InnerVolumeSpecName "kube-api-access-vwq8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:11:12.178367 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.178331    2556 generic.go:356] "Generic (PLEG): container finished" podID="ee3f87fc-8a34-4fa7-a23b-97268adcc33a" containerID="ffbded5fdb7978bd26e63ce6ee16349e029775d58d1c8a3957d10315eb4cc6b3" exitCode=0
Sep 09 16:11:12.178568 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.178378    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"
Sep 09 16:11:12.178568 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.178422    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn" event={"ID":"ee3f87fc-8a34-4fa7-a23b-97268adcc33a","Type":"ContainerDied","Data":"ffbded5fdb7978bd26e63ce6ee16349e029775d58d1c8a3957d10315eb4cc6b3"}
Sep 09 16:11:12.178568 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.178493    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn" event={"ID":"ee3f87fc-8a34-4fa7-a23b-97268adcc33a","Type":"ContainerDied","Data":"942d6237b85c86c0536d94b095f92d0eba6961119507a27067e089eb4bf8c989"}
Sep 09 16:11:12.178568 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.178519    2556 scope.go:117] "RemoveContainer" containerID="ffbded5fdb7978bd26e63ce6ee16349e029775d58d1c8a3957d10315eb4cc6b3"
Sep 09 16:11:12.183067 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.183043    2556 generic.go:356] "Generic (PLEG): container finished" podID="25aa6957-fb2d-40ef-8784-ae0e9ab28760" containerID="c2b6b326f45473d857bd2f3e681a76a27c26ef14136e3496babc6602693d9ac0" exitCode=0
Sep 09 16:11:12.183208 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.183100    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"
Sep 09 16:11:12.183208 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.183125    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7" event={"ID":"25aa6957-fb2d-40ef-8784-ae0e9ab28760","Type":"ContainerDied","Data":"c2b6b326f45473d857bd2f3e681a76a27c26ef14136e3496babc6602693d9ac0"}
Sep 09 16:11:12.183208 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.183172    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7" event={"ID":"25aa6957-fb2d-40ef-8784-ae0e9ab28760","Type":"ContainerDied","Data":"82dfc6720a925c21a17c5deda1ea87a0178f250c4e3d01924ad48e0fa63b1773"}
Sep 09 16:11:12.190795 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.190779    2556 scope.go:117] "RemoveContainer" containerID="ffbded5fdb7978bd26e63ce6ee16349e029775d58d1c8a3957d10315eb4cc6b3"
Sep 09 16:11:12.191083 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:11:12.191062    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffbded5fdb7978bd26e63ce6ee16349e029775d58d1c8a3957d10315eb4cc6b3\": container with ID starting with ffbded5fdb7978bd26e63ce6ee16349e029775d58d1c8a3957d10315eb4cc6b3 not found: ID does not exist" containerID="ffbded5fdb7978bd26e63ce6ee16349e029775d58d1c8a3957d10315eb4cc6b3"
Sep 09 16:11:12.191141 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.191092    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbded5fdb7978bd26e63ce6ee16349e029775d58d1c8a3957d10315eb4cc6b3"} err="failed to get container status \"ffbded5fdb7978bd26e63ce6ee16349e029775d58d1c8a3957d10315eb4cc6b3\": rpc error: code = NotFound desc = could not find container \"ffbded5fdb7978bd26e63ce6ee16349e029775d58d1c8a3957d10315eb4cc6b3\": container with ID starting with ffbded5fdb7978bd26e63ce6ee16349e029775d58d1c8a3957d10315eb4cc6b3 not found: ID does not exist"
Sep 09 16:11:12.191141 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.191110    2556 scope.go:117] "RemoveContainer" containerID="c2b6b326f45473d857bd2f3e681a76a27c26ef14136e3496babc6602693d9ac0"
Sep 09 16:11:12.198780 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.198766    2556 scope.go:117] "RemoveContainer" containerID="c2b6b326f45473d857bd2f3e681a76a27c26ef14136e3496babc6602693d9ac0"
Sep 09 16:11:12.199035 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:11:12.199018    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b6b326f45473d857bd2f3e681a76a27c26ef14136e3496babc6602693d9ac0\": container with ID starting with c2b6b326f45473d857bd2f3e681a76a27c26ef14136e3496babc6602693d9ac0 not found: ID does not exist" containerID="c2b6b326f45473d857bd2f3e681a76a27c26ef14136e3496babc6602693d9ac0"
Sep 09 16:11:12.199083 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.199039    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b6b326f45473d857bd2f3e681a76a27c26ef14136e3496babc6602693d9ac0"} err="failed to get container status \"c2b6b326f45473d857bd2f3e681a76a27c26ef14136e3496babc6602693d9ac0\": rpc error: code = NotFound desc = could not find container \"c2b6b326f45473d857bd2f3e681a76a27c26ef14136e3496babc6602693d9ac0\": container with ID starting with c2b6b326f45473d857bd2f3e681a76a27c26ef14136e3496babc6602693d9ac0 not found: ID does not exist"
Sep 09 16:11:12.203073 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.203052    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"]
Sep 09 16:11:12.210423 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.210400    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-58050ce1-786c65d7cc-67qdn"]
Sep 09 16:11:12.214639 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.214622    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwq8d\" (UniqueName: \"kubernetes.io/projected/ee3f87fc-8a34-4fa7-a23b-97268adcc33a-kube-api-access-vwq8d\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:12.214701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.214641    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3f87fc-8a34-4fa7-a23b-97268adcc33a-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:12.214701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.214652    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25aa6957-fb2d-40ef-8784-ae0e9ab28760-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:12.214701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.214662    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfkx9\" (UniqueName: \"kubernetes.io/projected/25aa6957-fb2d-40ef-8784-ae0e9ab28760-kube-api-access-bfkx9\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:12.221306 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.221281    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"]
Sep 09 16:11:12.225371 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:12.225348    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-58050ce1-95d5d4954-fxks7"]
Sep 09 16:11:13.766278 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:13.765381    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25aa6957-fb2d-40ef-8784-ae0e9ab28760" path="/var/lib/kubelet/pods/25aa6957-fb2d-40ef-8784-ae0e9ab28760/volumes"
Sep 09 16:11:13.766278 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:13.765941    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee3f87fc-8a34-4fa7-a23b-97268adcc33a" path="/var/lib/kubelet/pods/ee3f87fc-8a34-4fa7-a23b-97268adcc33a/volumes"
Sep 09 16:11:14.327836 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:14.327796    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:14.420189 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:14.420154    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:14.595115 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:14.595028    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:17.732542 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:17.732439    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:17.836789 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:17.836736    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:18.950035 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:18.949994    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"]
Sep 09 16:11:18.950717 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:18.950622    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee3f87fc-8a34-4fa7-a23b-97268adcc33a" containerName="payload-processing"
Sep 09 16:11:18.950717 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:18.950643    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee3f87fc-8a34-4fa7-a23b-97268adcc33a" containerName="payload-processing"
Sep 09 16:11:18.950717 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:18.950663    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25aa6957-fb2d-40ef-8784-ae0e9ab28760" containerName="payload-pre-processing"
Sep 09 16:11:18.950717 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:18.950671    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="25aa6957-fb2d-40ef-8784-ae0e9ab28760" containerName="payload-pre-processing"
Sep 09 16:11:18.950972 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:18.950804    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee3f87fc-8a34-4fa7-a23b-97268adcc33a" containerName="payload-processing"
Sep 09 16:11:18.950972 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:18.950823    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="25aa6957-fb2d-40ef-8784-ae0e9ab28760" containerName="payload-pre-processing"
Sep 09 16:11:18.955320 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:18.955298    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"
Sep 09 16:11:18.959319 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:18.959293    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2zvd\" (UniqueName: \"kubernetes.io/projected/14bf520b-fdab-4aaa-81a8-876c6424ca58-kube-api-access-f2zvd\") pod \"payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2\" (UID: \"14bf520b-fdab-4aaa-81a8-876c6424ca58\") " pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"
Sep 09 16:11:18.959421 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:18.959383    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14bf520b-fdab-4aaa-81a8-876c6424ca58-plugins-config-volume\") pod \"payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2\" (UID: \"14bf520b-fdab-4aaa-81a8-876c6424ca58\") " pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"
Sep 09 16:11:18.959802 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:18.959784    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f772da8e\""
Sep 09 16:11:18.959894 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:18.959817    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-f772da8e-dockercfg-6f95f\""
Sep 09 16:11:18.968402 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:18.968381    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"]
Sep 09 16:11:19.059829 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:19.059793    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14bf520b-fdab-4aaa-81a8-876c6424ca58-plugins-config-volume\") pod \"payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2\" (UID: \"14bf520b-fdab-4aaa-81a8-876c6424ca58\") " pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"
Sep 09 16:11:19.060022 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:19.059841    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2zvd\" (UniqueName: \"kubernetes.io/projected/14bf520b-fdab-4aaa-81a8-876c6424ca58-kube-api-access-f2zvd\") pod \"payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2\" (UID: \"14bf520b-fdab-4aaa-81a8-876c6424ca58\") " pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"
Sep 09 16:11:19.060535 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:19.060512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14bf520b-fdab-4aaa-81a8-876c6424ca58-plugins-config-volume\") pod \"payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2\" (UID: \"14bf520b-fdab-4aaa-81a8-876c6424ca58\") " pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"
Sep 09 16:11:19.071990 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:19.071958    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2zvd\" (UniqueName: \"kubernetes.io/projected/14bf520b-fdab-4aaa-81a8-876c6424ca58-kube-api-access-f2zvd\") pod \"payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2\" (UID: \"14bf520b-fdab-4aaa-81a8-876c6424ca58\") " pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"
Sep 09 16:11:19.265835 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:19.265745    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"
Sep 09 16:11:19.541359 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:19.538595    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"]
Sep 09 16:11:20.222000 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:20.221952    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:20.222655 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:20.222625    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2" event={"ID":"14bf520b-fdab-4aaa-81a8-876c6424ca58","Type":"ContainerStarted","Data":"279c4ceb891a00c61d64fbdc72f95b80bafb09fea9bd4416ff7bbb1e243a66e6"}
Sep 09 16:11:20.222792 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:20.222676    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2" event={"ID":"14bf520b-fdab-4aaa-81a8-876c6424ca58","Type":"ContainerStarted","Data":"3344e324aab2ed063431547e588fc8362975087f023377d9f901913dbe020b35"}
Sep 09 16:11:20.222792 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:20.222769    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"
Sep 09 16:11:20.244030 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:20.243978    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2" podStartSLOduration=2.243960914 podStartE2EDuration="2.243960914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:11:20.2435819 +0000 UTC m=+1185.110093119" watchObservedRunningTime="2026-09-09 16:11:20.243960914 +0000 UTC m=+1185.110472135"
Sep 09 16:11:20.320168 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:20.320126    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:27.943076 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:27.943037    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:31.229812 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:31.229778    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"
Sep 09 16:11:35.646458 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:35.646430    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 16:11:35.648480 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:35.648448    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 16:11:38.283250 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.283214    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:38.540729 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.540647    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"]
Sep 09 16:11:38.543873 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.543855    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.546588 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.546565    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7b8369\"/\"kube-root-ca.crt\""
Sep 09 16:11:38.547497 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.547459    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7b8369\"/\"default-dockercfg-jdzvk\""
Sep 09 16:11:38.547566 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.547533    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7b8369\"/\"test-model-7b8369-kserve-self-signed-certs\""
Sep 09 16:11:38.547566 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.547543    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7b8369\"/\"openshift-service-ca.crt\""
Sep 09 16:11:38.556110 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.556084    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"]
Sep 09 16:11:38.634687 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.634656    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-dshm\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.634687 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.634693    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvbq5\" (UniqueName: \"kubernetes.io/projected/96232610-6524-4e04-852d-886c6bdebf9e-kube-api-access-kvbq5\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.634909 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.634770    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-tmp-dir\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.634909 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.634823    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96232610-6524-4e04-852d-886c6bdebf9e-tls-certs\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.634909 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.634847    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-model-cache\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.634909 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.634903    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-home\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.735548 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.735497    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-dshm\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.735548 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.735553    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvbq5\" (UniqueName: \"kubernetes.io/projected/96232610-6524-4e04-852d-886c6bdebf9e-kube-api-access-kvbq5\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.735782 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.735599    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-tmp-dir\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.735782 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.735628    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96232610-6524-4e04-852d-886c6bdebf9e-tls-certs\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.735782 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.735657    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-model-cache\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.735782 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.735721    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-home\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.736066 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.736037    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-tmp-dir\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.736066 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.736054    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-home\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.736223 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.736093    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-model-cache\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.737952 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.737926    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-dshm\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.738135 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.738120    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96232610-6524-4e04-852d-886c6bdebf9e-tls-certs\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.745490 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.745450    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvbq5\" (UniqueName: \"kubernetes.io/projected/96232610-6524-4e04-852d-886c6bdebf9e-kube-api-access-kvbq5\") pod \"test-model-7b8369-kserve-7c69d94547-kxdvm\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") " pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:38.853748 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:38.853662    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:39.126184 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:39.126159    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"]
Sep 09 16:11:39.128753 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:11:39.128723    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96232610_6524_4e04_852d_886c6bdebf9e.slice/crio-4488310d243d34cc1808e53a7046ee9859d12ed56d250c871f2f179404b79420 WatchSource:0}: Error finding container 4488310d243d34cc1808e53a7046ee9859d12ed56d250c871f2f179404b79420: Status 404 returned error can't find the container with id 4488310d243d34cc1808e53a7046ee9859d12ed56d250c871f2f179404b79420
Sep 09 16:11:39.312286 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:39.312251    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm" event={"ID":"96232610-6524-4e04-852d-886c6bdebf9e","Type":"ContainerStarted","Data":"4815aad91d50eeb06e10465c4139a12cb32a00512c23eee01569dd50c35190bd"}
Sep 09 16:11:39.312700 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:39.312295    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm" event={"ID":"96232610-6524-4e04-852d-886c6bdebf9e","Type":"ContainerStarted","Data":"4488310d243d34cc1808e53a7046ee9859d12ed56d250c871f2f179404b79420"}
Sep 09 16:11:39.329606 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:39.329561    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm" podStartSLOduration=1.329546996 podStartE2EDuration="1.329546996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:11:39.328458658 +0000 UTC m=+1204.194969878" watchObservedRunningTime="2026-09-09 16:11:39.329546996 +0000 UTC m=+1204.196058216"
Sep 09 16:11:40.753126 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:40.753093    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:40.814953 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:40.814917    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:42.193254 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:42.193213    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:42.244898 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:42.244865    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"]
Sep 09 16:11:45.581897 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:45.581862    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:48.268197 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:48.263489    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:48.854756 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:48.854702    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:48.854946 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:48.854767    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:48.867771 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:48.867735    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:48.873781 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:48.873747    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"]
Sep 09 16:11:48.873965 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:48.873944    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2" podUID="14bf520b-fdab-4aaa-81a8-876c6424ca58" containerName="payload-processing" containerID="cri-o://279c4ceb891a00c61d64fbdc72f95b80bafb09fea9bd4416ff7bbb1e243a66e6" gracePeriod=30
Sep 09 16:11:48.936266 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:48.936229    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:49.029884 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.029843    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"]
Sep 09 16:11:49.030106 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.030079    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm" podUID="bf547411-e2ec-45ab-b55b-8d8930e411e9" containerName="payload-processing" containerID="cri-o://8667d0ec9eb229092437c49c18bd8019ab3fc42d46a3647dc46472d6f7179b58" gracePeriod=30
Sep 09 16:11:49.037207 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.037177    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"]
Sep 09 16:11:49.037930 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.037723    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt" podUID="426dd61b-5ccc-4666-954c-05a6924e5524" containerName="payload-pre-processing" containerID="cri-o://3a257a2af1d3257e9f3940df9dffa0e4b445fb0ccd2c09148f17e4c20071af5e" gracePeriod=30
Sep 09 16:11:49.163167 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.163139    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"
Sep 09 16:11:49.237783 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.237756    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2zvd\" (UniqueName: \"kubernetes.io/projected/14bf520b-fdab-4aaa-81a8-876c6424ca58-kube-api-access-f2zvd\") pod \"14bf520b-fdab-4aaa-81a8-876c6424ca58\" (UID: \"14bf520b-fdab-4aaa-81a8-876c6424ca58\") "
Sep 09 16:11:49.237935 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.237859    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14bf520b-fdab-4aaa-81a8-876c6424ca58-plugins-config-volume\") pod \"14bf520b-fdab-4aaa-81a8-876c6424ca58\" (UID: \"14bf520b-fdab-4aaa-81a8-876c6424ca58\") "
Sep 09 16:11:49.238291 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.238259    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14bf520b-fdab-4aaa-81a8-876c6424ca58-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14bf520b-fdab-4aaa-81a8-876c6424ca58" (UID: "14bf520b-fdab-4aaa-81a8-876c6424ca58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:11:49.240807 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.240781    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14bf520b-fdab-4aaa-81a8-876c6424ca58-kube-api-access-f2zvd" (OuterVolumeSpecName: "kube-api-access-f2zvd") pod "14bf520b-fdab-4aaa-81a8-876c6424ca58" (UID: "14bf520b-fdab-4aaa-81a8-876c6424ca58"). InnerVolumeSpecName "kube-api-access-f2zvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:11:49.299417 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.299378    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:49.308018 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.307992    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"
Sep 09 16:11:49.315270 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.315244    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"
Sep 09 16:11:49.338591 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.338563    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14bf520b-fdab-4aaa-81a8-876c6424ca58-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:49.338591 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.338588    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2zvd\" (UniqueName: \"kubernetes.io/projected/14bf520b-fdab-4aaa-81a8-876c6424ca58-kube-api-access-f2zvd\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:49.358364 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.358332    2556 generic.go:356] "Generic (PLEG): container finished" podID="bf547411-e2ec-45ab-b55b-8d8930e411e9" containerID="8667d0ec9eb229092437c49c18bd8019ab3fc42d46a3647dc46472d6f7179b58" exitCode=0
Sep 09 16:11:49.358585 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.358391    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"
Sep 09 16:11:49.358585 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.358408    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm" event={"ID":"bf547411-e2ec-45ab-b55b-8d8930e411e9","Type":"ContainerDied","Data":"8667d0ec9eb229092437c49c18bd8019ab3fc42d46a3647dc46472d6f7179b58"}
Sep 09 16:11:49.358585 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.358486    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm" event={"ID":"bf547411-e2ec-45ab-b55b-8d8930e411e9","Type":"ContainerDied","Data":"edc090bb58a3a882a4a9c1995b8b99e16fb60efdd30c91260421810b2059418d"}
Sep 09 16:11:49.358585 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.358489    2556 scope.go:117] "RemoveContainer" containerID="8667d0ec9eb229092437c49c18bd8019ab3fc42d46a3647dc46472d6f7179b58"
Sep 09 16:11:49.359753 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.359726    2556 generic.go:356] "Generic (PLEG): container finished" podID="426dd61b-5ccc-4666-954c-05a6924e5524" containerID="3a257a2af1d3257e9f3940df9dffa0e4b445fb0ccd2c09148f17e4c20071af5e" exitCode=0
Sep 09 16:11:49.359859 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.359765    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"
Sep 09 16:11:49.359859 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.359806    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt" event={"ID":"426dd61b-5ccc-4666-954c-05a6924e5524","Type":"ContainerDied","Data":"3a257a2af1d3257e9f3940df9dffa0e4b445fb0ccd2c09148f17e4c20071af5e"}
Sep 09 16:11:49.359859 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.359852    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt" event={"ID":"426dd61b-5ccc-4666-954c-05a6924e5524","Type":"ContainerDied","Data":"c961bb28f855b20227752549aa28bc9152e5fad986d61dc1f83bf8a72af9be9c"}
Sep 09 16:11:49.361170 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.361145    2556 generic.go:356] "Generic (PLEG): container finished" podID="14bf520b-fdab-4aaa-81a8-876c6424ca58" containerID="279c4ceb891a00c61d64fbdc72f95b80bafb09fea9bd4416ff7bbb1e243a66e6" exitCode=0
Sep 09 16:11:49.361275 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.361187    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"
Sep 09 16:11:49.361275 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.361217    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2" event={"ID":"14bf520b-fdab-4aaa-81a8-876c6424ca58","Type":"ContainerDied","Data":"279c4ceb891a00c61d64fbdc72f95b80bafb09fea9bd4416ff7bbb1e243a66e6"}
Sep 09 16:11:49.361275 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.361250    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2" event={"ID":"14bf520b-fdab-4aaa-81a8-876c6424ca58","Type":"ContainerDied","Data":"3344e324aab2ed063431547e588fc8362975087f023377d9f901913dbe020b35"}
Sep 09 16:11:49.364889 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.364849    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:49.368409 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.368391    2556 scope.go:117] "RemoveContainer" containerID="8667d0ec9eb229092437c49c18bd8019ab3fc42d46a3647dc46472d6f7179b58"
Sep 09 16:11:49.368710 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:11:49.368692    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8667d0ec9eb229092437c49c18bd8019ab3fc42d46a3647dc46472d6f7179b58\": container with ID starting with 8667d0ec9eb229092437c49c18bd8019ab3fc42d46a3647dc46472d6f7179b58 not found: ID does not exist" containerID="8667d0ec9eb229092437c49c18bd8019ab3fc42d46a3647dc46472d6f7179b58"
Sep 09 16:11:49.368793 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.368722    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8667d0ec9eb229092437c49c18bd8019ab3fc42d46a3647dc46472d6f7179b58"} err="failed to get container status \"8667d0ec9eb229092437c49c18bd8019ab3fc42d46a3647dc46472d6f7179b58\": rpc error: code = NotFound desc = could not find container \"8667d0ec9eb229092437c49c18bd8019ab3fc42d46a3647dc46472d6f7179b58\": container with ID starting with 8667d0ec9eb229092437c49c18bd8019ab3fc42d46a3647dc46472d6f7179b58 not found: ID does not exist"
Sep 09 16:11:49.368793 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.368744    2556 scope.go:117] "RemoveContainer" containerID="3a257a2af1d3257e9f3940df9dffa0e4b445fb0ccd2c09148f17e4c20071af5e"
Sep 09 16:11:49.374344 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.374305    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:11:49.379481 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.377932    2556 scope.go:117] "RemoveContainer" containerID="3a257a2af1d3257e9f3940df9dffa0e4b445fb0ccd2c09148f17e4c20071af5e"
Sep 09 16:11:49.379965 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:11:49.379946    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a257a2af1d3257e9f3940df9dffa0e4b445fb0ccd2c09148f17e4c20071af5e\": container with ID starting with 3a257a2af1d3257e9f3940df9dffa0e4b445fb0ccd2c09148f17e4c20071af5e not found: ID does not exist" containerID="3a257a2af1d3257e9f3940df9dffa0e4b445fb0ccd2c09148f17e4c20071af5e"
Sep 09 16:11:49.380088 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.379971    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a257a2af1d3257e9f3940df9dffa0e4b445fb0ccd2c09148f17e4c20071af5e"} err="failed to get container status \"3a257a2af1d3257e9f3940df9dffa0e4b445fb0ccd2c09148f17e4c20071af5e\": rpc error: code = NotFound desc = could not find container \"3a257a2af1d3257e9f3940df9dffa0e4b445fb0ccd2c09148f17e4c20071af5e\": container with ID starting with 3a257a2af1d3257e9f3940df9dffa0e4b445fb0ccd2c09148f17e4c20071af5e not found: ID does not exist"
Sep 09 16:11:49.380088 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.379989    2556 scope.go:117] "RemoveContainer" containerID="279c4ceb891a00c61d64fbdc72f95b80bafb09fea9bd4416ff7bbb1e243a66e6"
Sep 09 16:11:49.384179 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.384158    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"]
Sep 09 16:11:49.388060 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.388039    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f772da8e-6d5f6f4f8f-q94d2"]
Sep 09 16:11:49.388643 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.388629    2556 scope.go:117] "RemoveContainer" containerID="279c4ceb891a00c61d64fbdc72f95b80bafb09fea9bd4416ff7bbb1e243a66e6"
Sep 09 16:11:49.388881 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:11:49.388863    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"279c4ceb891a00c61d64fbdc72f95b80bafb09fea9bd4416ff7bbb1e243a66e6\": container with ID starting with 279c4ceb891a00c61d64fbdc72f95b80bafb09fea9bd4416ff7bbb1e243a66e6 not found: ID does not exist" containerID="279c4ceb891a00c61d64fbdc72f95b80bafb09fea9bd4416ff7bbb1e243a66e6"
Sep 09 16:11:49.388963 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.388890    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"279c4ceb891a00c61d64fbdc72f95b80bafb09fea9bd4416ff7bbb1e243a66e6"} err="failed to get container status \"279c4ceb891a00c61d64fbdc72f95b80bafb09fea9bd4416ff7bbb1e243a66e6\": rpc error: code = NotFound desc = could not find container \"279c4ceb891a00c61d64fbdc72f95b80bafb09fea9bd4416ff7bbb1e243a66e6\": container with ID starting with 279c4ceb891a00c61d64fbdc72f95b80bafb09fea9bd4416ff7bbb1e243a66e6 not found: ID does not exist"
Sep 09 16:11:49.439365 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.439341    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf547411-e2ec-45ab-b55b-8d8930e411e9-plugins-config-volume\") pod \"bf547411-e2ec-45ab-b55b-8d8930e411e9\" (UID: \"bf547411-e2ec-45ab-b55b-8d8930e411e9\") "
Sep 09 16:11:49.439550 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.439369    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xssm8\" (UniqueName: \"kubernetes.io/projected/bf547411-e2ec-45ab-b55b-8d8930e411e9-kube-api-access-xssm8\") pod \"bf547411-e2ec-45ab-b55b-8d8930e411e9\" (UID: \"bf547411-e2ec-45ab-b55b-8d8930e411e9\") "
Sep 09 16:11:49.439550 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.439403    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426dd61b-5ccc-4666-954c-05a6924e5524-plugins-config-volume\") pod \"426dd61b-5ccc-4666-954c-05a6924e5524\" (UID: \"426dd61b-5ccc-4666-954c-05a6924e5524\") "
Sep 09 16:11:49.439550 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.439490    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ttcv\" (UniqueName: \"kubernetes.io/projected/426dd61b-5ccc-4666-954c-05a6924e5524-kube-api-access-9ttcv\") pod \"426dd61b-5ccc-4666-954c-05a6924e5524\" (UID: \"426dd61b-5ccc-4666-954c-05a6924e5524\") "
Sep 09 16:11:49.439789 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.439761    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf547411-e2ec-45ab-b55b-8d8930e411e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf547411-e2ec-45ab-b55b-8d8930e411e9" (UID: "bf547411-e2ec-45ab-b55b-8d8930e411e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:11:49.439966 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.439948    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426dd61b-5ccc-4666-954c-05a6924e5524-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "426dd61b-5ccc-4666-954c-05a6924e5524" (UID: "426dd61b-5ccc-4666-954c-05a6924e5524"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:11:49.441728 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.441702    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426dd61b-5ccc-4666-954c-05a6924e5524-kube-api-access-9ttcv" (OuterVolumeSpecName: "kube-api-access-9ttcv") pod "426dd61b-5ccc-4666-954c-05a6924e5524" (UID: "426dd61b-5ccc-4666-954c-05a6924e5524"). InnerVolumeSpecName "kube-api-access-9ttcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:11:49.441868 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.441848    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf547411-e2ec-45ab-b55b-8d8930e411e9-kube-api-access-xssm8" (OuterVolumeSpecName: "kube-api-access-xssm8") pod "bf547411-e2ec-45ab-b55b-8d8930e411e9" (UID: "bf547411-e2ec-45ab-b55b-8d8930e411e9"). InnerVolumeSpecName "kube-api-access-xssm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:11:49.540753 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.540711    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf547411-e2ec-45ab-b55b-8d8930e411e9-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:49.540753 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.540747    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xssm8\" (UniqueName: \"kubernetes.io/projected/bf547411-e2ec-45ab-b55b-8d8930e411e9-kube-api-access-xssm8\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:49.540753 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.540760    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426dd61b-5ccc-4666-954c-05a6924e5524-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:49.540984 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.540772    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ttcv\" (UniqueName: \"kubernetes.io/projected/426dd61b-5ccc-4666-954c-05a6924e5524-kube-api-access-9ttcv\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:49.651886 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.651840    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:49.682763 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.682736    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"]
Sep 09 16:11:49.686543 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.686519    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a09c2a-7d78799598-srjtt"]
Sep 09 16:11:49.699954 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.699880    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"]
Sep 09 16:11:49.703929 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.703907    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a09c2a-6c59646f7c-vmdjm"]
Sep 09 16:11:49.765017 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.764981    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14bf520b-fdab-4aaa-81a8-876c6424ca58" path="/var/lib/kubelet/pods/14bf520b-fdab-4aaa-81a8-876c6424ca58/volumes"
Sep 09 16:11:49.765342 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.765329    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="426dd61b-5ccc-4666-954c-05a6924e5524" path="/var/lib/kubelet/pods/426dd61b-5ccc-4666-954c-05a6924e5524/volumes"
Sep 09 16:11:49.765682 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.765669    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf547411-e2ec-45ab-b55b-8d8930e411e9" path="/var/lib/kubelet/pods/bf547411-e2ec-45ab-b55b-8d8930e411e9/volumes"
Sep 09 16:11:49.961271 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:49.961178    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:50.051692 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:50.051657    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:50.452373 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:50.452058    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:51.629243 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:51.629195    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:54.729689 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:54.729654    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:57.263843 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.263801    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v" podUID="3d73e719-8b8d-4a10-9431-13d272a667ab" containerName="main" containerID="cri-o://1da76417bde985ceb87bef12b46c1a338b78c41af88008248fe76dd80eade84a" gracePeriod=45
Sep 09 16:11:57.401651 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.401619    2556 generic.go:356] "Generic (PLEG): container finished" podID="3d73e719-8b8d-4a10-9431-13d272a667ab" containerID="1da76417bde985ceb87bef12b46c1a338b78c41af88008248fe76dd80eade84a" exitCode=0
Sep 09 16:11:57.401796 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.401690    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v" event={"ID":"3d73e719-8b8d-4a10-9431-13d272a667ab","Type":"ContainerDied","Data":"1da76417bde985ceb87bef12b46c1a338b78c41af88008248fe76dd80eade84a"}
Sep 09 16:11:57.512581 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.512556    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:11:57.607559 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.607481    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d73e719-8b8d-4a10-9431-13d272a667ab-tls-certs\") pod \"3d73e719-8b8d-4a10-9431-13d272a667ab\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") "
Sep 09 16:11:57.607559 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.607549    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-home\") pod \"3d73e719-8b8d-4a10-9431-13d272a667ab\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") "
Sep 09 16:11:57.607733 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.607688    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-dshm\") pod \"3d73e719-8b8d-4a10-9431-13d272a667ab\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") "
Sep 09 16:11:57.607780 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.607733    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-tmp-dir\") pod \"3d73e719-8b8d-4a10-9431-13d272a667ab\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") "
Sep 09 16:11:57.607780 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.607740    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-home" (OuterVolumeSpecName: "home") pod "3d73e719-8b8d-4a10-9431-13d272a667ab" (UID: "3d73e719-8b8d-4a10-9431-13d272a667ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:11:57.607873 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.607783    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkcqw\" (UniqueName: \"kubernetes.io/projected/3d73e719-8b8d-4a10-9431-13d272a667ab-kube-api-access-gkcqw\") pod \"3d73e719-8b8d-4a10-9431-13d272a667ab\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") "
Sep 09 16:11:57.607873 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.607805    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-model-cache\") pod \"3d73e719-8b8d-4a10-9431-13d272a667ab\" (UID: \"3d73e719-8b8d-4a10-9431-13d272a667ab\") "
Sep 09 16:11:57.607982 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.607925    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d73e719-8b8d-4a10-9431-13d272a667ab" (UID: "3d73e719-8b8d-4a10-9431-13d272a667ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:11:57.608089 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.608065    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-home\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:57.608089 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.608089    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-tmp-dir\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:57.608278 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.608120    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d73e719-8b8d-4a10-9431-13d272a667ab" (UID: "3d73e719-8b8d-4a10-9431-13d272a667ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:11:57.609856 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.609825    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d73e719-8b8d-4a10-9431-13d272a667ab-kube-api-access-gkcqw" (OuterVolumeSpecName: "kube-api-access-gkcqw") pod "3d73e719-8b8d-4a10-9431-13d272a667ab" (UID: "3d73e719-8b8d-4a10-9431-13d272a667ab"). InnerVolumeSpecName "kube-api-access-gkcqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:11:57.609933 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.609864    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d73e719-8b8d-4a10-9431-13d272a667ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d73e719-8b8d-4a10-9431-13d272a667ab" (UID: "3d73e719-8b8d-4a10-9431-13d272a667ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:11:57.609933 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.609867    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-dshm" (OuterVolumeSpecName: "dshm") pod "3d73e719-8b8d-4a10-9431-13d272a667ab" (UID: "3d73e719-8b8d-4a10-9431-13d272a667ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:11:57.679501 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.679444    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:57.708515 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.708491    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkcqw\" (UniqueName: \"kubernetes.io/projected/3d73e719-8b8d-4a10-9431-13d272a667ab-kube-api-access-gkcqw\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:57.708515 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.708514    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-model-cache\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:57.708691 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.708525    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d73e719-8b8d-4a10-9431-13d272a667ab-tls-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:57.708691 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.708533    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d73e719-8b8d-4a10-9431-13d272a667ab-dshm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:11:57.801113 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.801075    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:57.912886 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:57.912803    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:58.245771 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:58.245735    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:58.402807 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:58.402764    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:11:58.407256 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:58.407229    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"
Sep 09 16:11:58.407395 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:58.407259    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v" event={"ID":"3d73e719-8b8d-4a10-9431-13d272a667ab","Type":"ContainerDied","Data":"0facb051afa04be2422a15830bec30deb0d260dac54bebe6edeabfb0c53b3b30"}
Sep 09 16:11:58.407395 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:58.407305    2556 scope.go:117] "RemoveContainer" containerID="1da76417bde985ceb87bef12b46c1a338b78c41af88008248fe76dd80eade84a"
Sep 09 16:11:58.422673 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:58.422650    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"]
Sep 09 16:11:58.427945 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:58.427925    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-049e4f89-kserve-7f7cfd8c8b-xcm8v"]
Sep 09 16:11:59.763092 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:11:59.763049    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d73e719-8b8d-4a10-9431-13d272a667ab" path="/var/lib/kubelet/pods/3d73e719-8b8d-4a10-9431-13d272a667ab/volumes"
Sep 09 16:12:10.938296 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.938259    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"]
Sep 09 16:12:10.939541 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.939514    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf547411-e2ec-45ab-b55b-8d8930e411e9" containerName="payload-processing"
Sep 09 16:12:10.939708 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.939696    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf547411-e2ec-45ab-b55b-8d8930e411e9" containerName="payload-processing"
Sep 09 16:12:10.939809 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.939799    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14bf520b-fdab-4aaa-81a8-876c6424ca58" containerName="payload-processing"
Sep 09 16:12:10.939887 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.939879    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="14bf520b-fdab-4aaa-81a8-876c6424ca58" containerName="payload-processing"
Sep 09 16:12:10.939982 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.939973    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d73e719-8b8d-4a10-9431-13d272a667ab" containerName="main"
Sep 09 16:12:10.940052 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.940044    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d73e719-8b8d-4a10-9431-13d272a667ab" containerName="main"
Sep 09 16:12:10.940142 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.940132    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426dd61b-5ccc-4666-954c-05a6924e5524" containerName="payload-pre-processing"
Sep 09 16:12:10.940213 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.940206    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="426dd61b-5ccc-4666-954c-05a6924e5524" containerName="payload-pre-processing"
Sep 09 16:12:10.940485 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.940453    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="14bf520b-fdab-4aaa-81a8-876c6424ca58" containerName="payload-processing"
Sep 09 16:12:10.940616 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.940603    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="426dd61b-5ccc-4666-954c-05a6924e5524" containerName="payload-pre-processing"
Sep 09 16:12:10.940700 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.940691    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf547411-e2ec-45ab-b55b-8d8930e411e9" containerName="payload-processing"
Sep 09 16:12:10.940788 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.940779    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d73e719-8b8d-4a10-9431-13d272a667ab" containerName="main"
Sep 09 16:12:10.944829 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.944802    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:10.948144 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.948119    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c23ff1\"/\"kube-root-ca.crt\""
Sep 09 16:12:10.948250 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.948232    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c23ff1\"/\"default-dockercfg-n72vq\""
Sep 09 16:12:10.948347 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.948323    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c23ff1\"/\"test-model-c23ff1-kserve-self-signed-certs\""
Sep 09 16:12:10.948410 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.948350    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c23ff1\"/\"openshift-service-ca.crt\""
Sep 09 16:12:10.951040 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.951018    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"]
Sep 09 16:12:10.967735 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:10.967705    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:11.018044 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.018010    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-home\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.018191 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.018049    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-tmp-dir\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.018191 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.018074    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlzjw\" (UniqueName: \"kubernetes.io/projected/75e9310c-aa7c-4894-a818-0d4c0adee060-kube-api-access-xlzjw\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.018191 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.018098    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-model-cache\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.018191 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.018115    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-dshm\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.018331 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.018185    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e9310c-aa7c-4894-a818-0d4c0adee060-tls-certs\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.118864 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.118827    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlzjw\" (UniqueName: \"kubernetes.io/projected/75e9310c-aa7c-4894-a818-0d4c0adee060-kube-api-access-xlzjw\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.118864 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.118867    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-model-cache\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.119062 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.118885    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-dshm\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.119062 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.118911    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e9310c-aa7c-4894-a818-0d4c0adee060-tls-certs\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.119062 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.118999    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-home\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.119062 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.119028    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-tmp-dir\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.119293 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.119234    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-model-cache\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.119378 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.119344    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-tmp-dir\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.119442 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.119401    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-home\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.121257 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.121231    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-dshm\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.121416 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.121400    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e9310c-aa7c-4894-a818-0d4c0adee060-tls-certs\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.126992 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.126971    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlzjw\" (UniqueName: \"kubernetes.io/projected/75e9310c-aa7c-4894-a818-0d4c0adee060-kube-api-access-xlzjw\") pod \"test-model-c23ff1-kserve-5f9bb84f87-mwtq9\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") " pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.255162 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.255070    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:11.525619 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:11.525592    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"]
Sep 09 16:12:11.527960 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:12:11.527932    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75e9310c_aa7c_4894_a818_0d4c0adee060.slice/crio-4fe0a1b82e8df9df77fd069e2cf31d5967d2e99592b13f887182648e481d1164 WatchSource:0}: Error finding container 4fe0a1b82e8df9df77fd069e2cf31d5967d2e99592b13f887182648e481d1164: Status 404 returned error can't find the container with id 4fe0a1b82e8df9df77fd069e2cf31d5967d2e99592b13f887182648e481d1164
Sep 09 16:12:12.467553 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:12.467516    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9" event={"ID":"75e9310c-aa7c-4894-a818-0d4c0adee060","Type":"ContainerStarted","Data":"a804ca0696eab6aa0e9dac020d0e0d9c72d5abb1ff7db459c4baf3522932f167"}
Sep 09 16:12:12.467943 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:12.467562    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9" event={"ID":"75e9310c-aa7c-4894-a818-0d4c0adee060","Type":"ContainerStarted","Data":"4fe0a1b82e8df9df77fd069e2cf31d5967d2e99592b13f887182648e481d1164"}
Sep 09 16:12:21.255427 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:21.255383    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:21.255856 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:21.255450    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:21.267895 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:21.267866    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:21.284417 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:21.284369    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9" podStartSLOduration=11.284354819 podStartE2EDuration="11.284354819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:12:12.482254287 +0000 UTC m=+1237.348765508" watchObservedRunningTime="2026-09-09 16:12:21.284354819 +0000 UTC m=+1246.150866062"
Sep 09 16:12:21.517287 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:21.517202    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:12:24.399784 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:24.399750    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:30.137767 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:30.137667    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:30.295617 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:30.295582    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:30.397033 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:30.396932    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:32.086804 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:32.086766    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:33.668623 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:33.668589    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:33.769515 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:33.769478    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:36.219247 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:36.219202    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:36.310661 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:36.310615    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:37.367020 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:37.366988    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:37.962081 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:37.962048    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"]
Sep 09 16:12:37.966261 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:37.966240    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"
Sep 09 16:12:37.968627 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:37.968605    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-bfb318-dockercfg-89blk\""
Sep 09 16:12:37.968750 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:37.968658    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-bfb318\""
Sep 09 16:12:37.974515 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:37.974492    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"]
Sep 09 16:12:38.047073 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:38.047040    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405ef3bc-1a82-4fdf-85fe-87bee2cde743-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx\" (UID: \"405ef3bc-1a82-4fdf-85fe-87bee2cde743\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"
Sep 09 16:12:38.047255 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:38.047107    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpw54\" (UniqueName: \"kubernetes.io/projected/405ef3bc-1a82-4fdf-85fe-87bee2cde743-kube-api-access-rpw54\") pod \"payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx\" (UID: \"405ef3bc-1a82-4fdf-85fe-87bee2cde743\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"
Sep 09 16:12:38.147815 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:38.147780    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405ef3bc-1a82-4fdf-85fe-87bee2cde743-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx\" (UID: \"405ef3bc-1a82-4fdf-85fe-87bee2cde743\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"
Sep 09 16:12:38.147965 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:38.147831    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpw54\" (UniqueName: \"kubernetes.io/projected/405ef3bc-1a82-4fdf-85fe-87bee2cde743-kube-api-access-rpw54\") pod \"payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx\" (UID: \"405ef3bc-1a82-4fdf-85fe-87bee2cde743\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"
Sep 09 16:12:38.148519 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:38.148500    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405ef3bc-1a82-4fdf-85fe-87bee2cde743-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx\" (UID: \"405ef3bc-1a82-4fdf-85fe-87bee2cde743\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"
Sep 09 16:12:38.156422 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:38.156398    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpw54\" (UniqueName: \"kubernetes.io/projected/405ef3bc-1a82-4fdf-85fe-87bee2cde743-kube-api-access-rpw54\") pod \"payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx\" (UID: \"405ef3bc-1a82-4fdf-85fe-87bee2cde743\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"
Sep 09 16:12:38.277295 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:38.277205    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"
Sep 09 16:12:38.315874 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:38.315832    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:38.416329 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:38.416287    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:38.545357 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:38.545306    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"]
Sep 09 16:12:38.547526 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:12:38.547499    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405ef3bc_1a82_4fdf_85fe_87bee2cde743.slice/crio-f7acf0f0419921d78456fb59a4cdaf914b01e84830639598cf8f43f2d6f18178 WatchSource:0}: Error finding container f7acf0f0419921d78456fb59a4cdaf914b01e84830639598cf8f43f2d6f18178: Status 404 returned error can't find the container with id f7acf0f0419921d78456fb59a4cdaf914b01e84830639598cf8f43f2d6f18178
Sep 09 16:12:38.581922 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:38.581895    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx" event={"ID":"405ef3bc-1a82-4fdf-85fe-87bee2cde743","Type":"ContainerStarted","Data":"f7acf0f0419921d78456fb59a4cdaf914b01e84830639598cf8f43f2d6f18178"}
Sep 09 16:12:39.588913 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:39.588877    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx" event={"ID":"405ef3bc-1a82-4fdf-85fe-87bee2cde743","Type":"ContainerStarted","Data":"a2ac0528cc0efc9271ceac53fe2a5441352b05924802b3254472faaba11d712c"}
Sep 09 16:12:39.589315 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:39.589001    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"
Sep 09 16:12:39.608341 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:39.608287    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx" podStartSLOduration=2.608271688 podStartE2EDuration="2.608271688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:12:39.60580223 +0000 UTC m=+1264.472313450" watchObservedRunningTime="2026-09-09 16:12:39.608271688 +0000 UTC m=+1264.474782908"
Sep 09 16:12:46.194327 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:46.194291    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:46.959007 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:46.958969    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:47.008928 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:47.008891    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:50.595434 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:50.595401    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"
Sep 09 16:12:51.914506 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:51.914452    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:52.029239 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:52.029197    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:12:52.119745 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:12:52.119712    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:01.218692 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:01.218648    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:02.620134 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:02.620097    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:03.398528 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:03.398488    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:08.873102 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:08.873068    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:09.440358 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:09.440322    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:09.552708 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:09.552664    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:09.648477 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:09.648441    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:09.746967 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:09.746866    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:10.407429 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:10.407394    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:10.581969 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:10.581930    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:10.676127 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:10.676080    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:12.132794 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:12.132754    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:18.150132 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:18.150095    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:18.214871 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:18.214836    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:18.829714 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:18.829672    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:24.670915 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:24.670880    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"]
Sep 09 16:13:24.691651 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:24.691615    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:24.949228 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:24.949193    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:25.050359 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:25.050320    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:25.177243 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:25.177201    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:25.496855 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:25.496813    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:25.783916 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:25.783830    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:25.853741 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:25.853690    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:25.954558 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:25.954525    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bfb318-mggxz"]
Sep 09 16:13:25.957861 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:25.957844    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bfb318-mggxz"
Sep 09 16:13:25.960064 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:25.960045    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\""
Sep 09 16:13:25.960327 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:25.960301    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:13:25.965681 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:25.965658    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bfb318-mggxz"]
Sep 09 16:13:26.063227 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:26.063129    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ad114a-db9b-4b2e-93a6-54d580c9fea9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-bfb318-mggxz\" (UID: \"12ad114a-db9b-4b2e-93a6-54d580c9fea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bfb318-mggxz"
Sep 09 16:13:26.164453 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:26.164417    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ad114a-db9b-4b2e-93a6-54d580c9fea9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-bfb318-mggxz\" (UID: \"12ad114a-db9b-4b2e-93a6-54d580c9fea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bfb318-mggxz"
Sep 09 16:13:26.165083 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:26.165060    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ad114a-db9b-4b2e-93a6-54d580c9fea9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-bfb318-mggxz\" (UID: \"12ad114a-db9b-4b2e-93a6-54d580c9fea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bfb318-mggxz"
Sep 09 16:13:26.269511 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:26.269436    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bfb318-mggxz"
Sep 09 16:13:26.537992 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:26.537950    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bfb318-mggxz"]
Sep 09 16:13:26.540996 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:13:26.540962    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ad114a_db9b_4b2e_93a6_54d580c9fea9.slice/crio-753baeb0aa54f686eea8d38ba0c39bc8cfaccf5b4662ceb688bf8289e9c3d153 WatchSource:0}: Error finding container 753baeb0aa54f686eea8d38ba0c39bc8cfaccf5b4662ceb688bf8289e9c3d153: Status 404 returned error can't find the container with id 753baeb0aa54f686eea8d38ba0c39bc8cfaccf5b4662ceb688bf8289e9c3d153
Sep 09 16:13:26.793378 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:26.793344    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bfb318-mggxz" event={"ID":"12ad114a-db9b-4b2e-93a6-54d580c9fea9","Type":"ContainerStarted","Data":"753baeb0aa54f686eea8d38ba0c39bc8cfaccf5b4662ceb688bf8289e9c3d153"}
Sep 09 16:13:27.036531 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:27.036496    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"]
Sep 09 16:13:27.064410 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:27.064330    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:27.297926 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:27.297890    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:28.757683 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:28.757645    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:28.953014 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:28.952911    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:29.814621 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:29.814589    2556 generic.go:356] "Generic (PLEG): container finished" podID="12ad114a-db9b-4b2e-93a6-54d580c9fea9" containerID="e72540393809f9599b16e4ed880381a476e4a8873090510d4300269a4d897ed7" exitCode=0
Sep 09 16:13:29.815055 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:29.814634    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bfb318-mggxz" event={"ID":"12ad114a-db9b-4b2e-93a6-54d580c9fea9","Type":"ContainerDied","Data":"e72540393809f9599b16e4ed880381a476e4a8873090510d4300269a4d897ed7"}
Sep 09 16:13:29.832914 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:29.832885    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:29.929259 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:29.929229    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:30.078267 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.078180    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:30.435828 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.435787    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:30.519376 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.519340    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:30.523511 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.523488    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"]
Sep 09 16:13:30.526772 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.526755    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"
Sep 09 16:13:30.529193 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.529170    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-677907eb\""
Sep 09 16:13:30.536727 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.536703    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"]
Sep 09 16:13:30.561983 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.561954    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"]
Sep 09 16:13:30.565344 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.565326    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"
Sep 09 16:13:30.567564 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.567546    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-677907eb-dockercfg-dlmhw\""
Sep 09 16:13:30.575227 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.575206    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"]
Sep 09 16:13:30.602813 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.602779    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbc9t\" (UniqueName: \"kubernetes.io/projected/7360aac7-5a06-4a34-bfa7-59fc0a98471e-kube-api-access-mbc9t\") pod \"payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg\" (UID: \"7360aac7-5a06-4a34-bfa7-59fc0a98471e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"
Sep 09 16:13:30.602969 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.602826    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7360aac7-5a06-4a34-bfa7-59fc0a98471e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg\" (UID: \"7360aac7-5a06-4a34-bfa7-59fc0a98471e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"
Sep 09 16:13:30.704114 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.704028    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbc9t\" (UniqueName: \"kubernetes.io/projected/7360aac7-5a06-4a34-bfa7-59fc0a98471e-kube-api-access-mbc9t\") pod \"payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg\" (UID: \"7360aac7-5a06-4a34-bfa7-59fc0a98471e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"
Sep 09 16:13:30.704114 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.704104    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c28mw\" (UniqueName: \"kubernetes.io/projected/ae7a5564-8f34-45e0-9ade-d2475ba94da3-kube-api-access-c28mw\") pod \"payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg\" (UID: \"ae7a5564-8f34-45e0-9ade-d2475ba94da3\") " pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"
Sep 09 16:13:30.704355 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.704143    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7360aac7-5a06-4a34-bfa7-59fc0a98471e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg\" (UID: \"7360aac7-5a06-4a34-bfa7-59fc0a98471e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"
Sep 09 16:13:30.704355 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.704228    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7a5564-8f34-45e0-9ade-d2475ba94da3-plugins-config-volume\") pod \"payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg\" (UID: \"ae7a5564-8f34-45e0-9ade-d2475ba94da3\") " pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"
Sep 09 16:13:30.704926 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.704907    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7360aac7-5a06-4a34-bfa7-59fc0a98471e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg\" (UID: \"7360aac7-5a06-4a34-bfa7-59fc0a98471e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"
Sep 09 16:13:30.711258 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.711235    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbc9t\" (UniqueName: \"kubernetes.io/projected/7360aac7-5a06-4a34-bfa7-59fc0a98471e-kube-api-access-mbc9t\") pod \"payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg\" (UID: \"7360aac7-5a06-4a34-bfa7-59fc0a98471e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"
Sep 09 16:13:30.805250 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.805217    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c28mw\" (UniqueName: \"kubernetes.io/projected/ae7a5564-8f34-45e0-9ade-d2475ba94da3-kube-api-access-c28mw\") pod \"payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg\" (UID: \"ae7a5564-8f34-45e0-9ade-d2475ba94da3\") " pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"
Sep 09 16:13:30.805420 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.805265    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7a5564-8f34-45e0-9ade-d2475ba94da3-plugins-config-volume\") pod \"payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg\" (UID: \"ae7a5564-8f34-45e0-9ade-d2475ba94da3\") " pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"
Sep 09 16:13:30.805947 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.805926    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7a5564-8f34-45e0-9ade-d2475ba94da3-plugins-config-volume\") pod \"payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg\" (UID: \"ae7a5564-8f34-45e0-9ade-d2475ba94da3\") " pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"
Sep 09 16:13:30.813320 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.813293    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c28mw\" (UniqueName: \"kubernetes.io/projected/ae7a5564-8f34-45e0-9ade-d2475ba94da3-kube-api-access-c28mw\") pod \"payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg\" (UID: \"ae7a5564-8f34-45e0-9ade-d2475ba94da3\") " pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"
Sep 09 16:13:30.837314 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.837283    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"
Sep 09 16:13:30.876137 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.875683    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"
Sep 09 16:13:30.946574 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:30.946549    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bfb318-mggxz"
Sep 09 16:13:31.007792 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.007709    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ad114a-db9b-4b2e-93a6-54d580c9fea9-maas-api-service-ca\") pod \"12ad114a-db9b-4b2e-93a6-54d580c9fea9\" (UID: \"12ad114a-db9b-4b2e-93a6-54d580c9fea9\") "
Sep 09 16:13:31.008105 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.008080    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ad114a-db9b-4b2e-93a6-54d580c9fea9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12ad114a-db9b-4b2e-93a6-54d580c9fea9" (UID: "12ad114a-db9b-4b2e-93a6-54d580c9fea9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:13:31.108748 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.108716    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ad114a-db9b-4b2e-93a6-54d580c9fea9-maas-api-service-ca\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:31.118567 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.118540    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"]
Sep 09 16:13:31.120404 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:13:31.120378    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7360aac7_5a06_4a34_bfa7_59fc0a98471e.slice/crio-7caa706d6d6597d19e5bce6d1470a335399ad4c6fca46220554a448795506627 WatchSource:0}: Error finding container 7caa706d6d6597d19e5bce6d1470a335399ad4c6fca46220554a448795506627: Status 404 returned error can't find the container with id 7caa706d6d6597d19e5bce6d1470a335399ad4c6fca46220554a448795506627
Sep 09 16:13:31.166561 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.166537    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"]
Sep 09 16:13:31.169201 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:13:31.169179    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7a5564_8f34_45e0_9ade_d2475ba94da3.slice/crio-98597fe14fb8c39b1ac0bf64e2fbd5dbe766f0f7bf01ddbb0daf7905481dc15f WatchSource:0}: Error finding container 98597fe14fb8c39b1ac0bf64e2fbd5dbe766f0f7bf01ddbb0daf7905481dc15f: Status 404 returned error can't find the container with id 98597fe14fb8c39b1ac0bf64e2fbd5dbe766f0f7bf01ddbb0daf7905481dc15f
Sep 09 16:13:31.826140 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.826112    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bfb318-mggxz"
Sep 09 16:13:31.826295 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.826110    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bfb318-mggxz" event={"ID":"12ad114a-db9b-4b2e-93a6-54d580c9fea9","Type":"ContainerDied","Data":"753baeb0aa54f686eea8d38ba0c39bc8cfaccf5b4662ceb688bf8289e9c3d153"}
Sep 09 16:13:31.826295 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.826264    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="753baeb0aa54f686eea8d38ba0c39bc8cfaccf5b4662ceb688bf8289e9c3d153"
Sep 09 16:13:31.828288 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.828122    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg" event={"ID":"ae7a5564-8f34-45e0-9ade-d2475ba94da3","Type":"ContainerStarted","Data":"e4ace5c03d1b03f08d652223a0db576fb61a28346b969fd7f8c49492b5534415"}
Sep 09 16:13:31.828288 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.828174    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg" event={"ID":"ae7a5564-8f34-45e0-9ade-d2475ba94da3","Type":"ContainerStarted","Data":"98597fe14fb8c39b1ac0bf64e2fbd5dbe766f0f7bf01ddbb0daf7905481dc15f"}
Sep 09 16:13:31.828544 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.828338    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"
Sep 09 16:13:31.829972 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.829949    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg" event={"ID":"7360aac7-5a06-4a34-bfa7-59fc0a98471e","Type":"ContainerStarted","Data":"3631926ebb412055bb728c64c1a6f7c8d6ca68f891374726a8d5cf8e33f22eab"}
Sep 09 16:13:31.830078 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.829992    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg" event={"ID":"7360aac7-5a06-4a34-bfa7-59fc0a98471e","Type":"ContainerStarted","Data":"7caa706d6d6597d19e5bce6d1470a335399ad4c6fca46220554a448795506627"}
Sep 09 16:13:31.830078 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.830020    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"
Sep 09 16:13:31.844454 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.844391    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg" podStartSLOduration=1.844371372 podStartE2EDuration="1.844371372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:13:31.843896111 +0000 UTC m=+1316.710407332" watchObservedRunningTime="2026-09-09 16:13:31.844371372 +0000 UTC m=+1316.710882590"
Sep 09 16:13:31.867883 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:31.867826    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg" podStartSLOduration=1.867806314 podStartE2EDuration="1.867806314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:13:31.863915849 +0000 UTC m=+1316.730427070" watchObservedRunningTime="2026-09-09 16:13:31.867806314 +0000 UTC m=+1316.734317536"
Sep 09 16:13:36.102095 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:36.102057    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:36.898844 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:36.898810    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:36.927404 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:36.927370    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"]
Sep 09 16:13:36.927702 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:36.927649    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx" podUID="405ef3bc-1a82-4fdf-85fe-87bee2cde743" containerName="payload-pre-processing" containerID="cri-o://a2ac0528cc0efc9271ceac53fe2a5441352b05924802b3254472faaba11d712c" gracePeriod=30
Sep 09 16:13:36.967646 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:36.967607    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:37.169952 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.169924    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bfb318-mggxz"]
Sep 09 16:13:37.170530 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.170507    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"
Sep 09 16:13:37.172975 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.172951    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bfb318-mggxz"]
Sep 09 16:13:37.263488 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.263446    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpw54\" (UniqueName: \"kubernetes.io/projected/405ef3bc-1a82-4fdf-85fe-87bee2cde743-kube-api-access-rpw54\") pod \"405ef3bc-1a82-4fdf-85fe-87bee2cde743\" (UID: \"405ef3bc-1a82-4fdf-85fe-87bee2cde743\") "
Sep 09 16:13:37.263672 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.263538    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405ef3bc-1a82-4fdf-85fe-87bee2cde743-plugins-config-volume\") pod \"405ef3bc-1a82-4fdf-85fe-87bee2cde743\" (UID: \"405ef3bc-1a82-4fdf-85fe-87bee2cde743\") "
Sep 09 16:13:37.263925 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.263901    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405ef3bc-1a82-4fdf-85fe-87bee2cde743-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "405ef3bc-1a82-4fdf-85fe-87bee2cde743" (UID: "405ef3bc-1a82-4fdf-85fe-87bee2cde743"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:13:37.265550 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.265520    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405ef3bc-1a82-4fdf-85fe-87bee2cde743-kube-api-access-rpw54" (OuterVolumeSpecName: "kube-api-access-rpw54") pod "405ef3bc-1a82-4fdf-85fe-87bee2cde743" (UID: "405ef3bc-1a82-4fdf-85fe-87bee2cde743"). InnerVolumeSpecName "kube-api-access-rpw54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:13:37.364364 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.364330    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpw54\" (UniqueName: \"kubernetes.io/projected/405ef3bc-1a82-4fdf-85fe-87bee2cde743-kube-api-access-rpw54\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:37.364364 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.364360    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405ef3bc-1a82-4fdf-85fe-87bee2cde743-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:37.763588 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.763549    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12ad114a-db9b-4b2e-93a6-54d580c9fea9" path="/var/lib/kubelet/pods/12ad114a-db9b-4b2e-93a6-54d580c9fea9/volumes"
Sep 09 16:13:37.860933 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.860896    2556 generic.go:356] "Generic (PLEG): container finished" podID="405ef3bc-1a82-4fdf-85fe-87bee2cde743" containerID="a2ac0528cc0efc9271ceac53fe2a5441352b05924802b3254472faaba11d712c" exitCode=0
Sep 09 16:13:37.861095 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.860944    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"
Sep 09 16:13:37.861095 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.860986    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx" event={"ID":"405ef3bc-1a82-4fdf-85fe-87bee2cde743","Type":"ContainerDied","Data":"a2ac0528cc0efc9271ceac53fe2a5441352b05924802b3254472faaba11d712c"}
Sep 09 16:13:37.861095 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.861034    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx" event={"ID":"405ef3bc-1a82-4fdf-85fe-87bee2cde743","Type":"ContainerDied","Data":"f7acf0f0419921d78456fb59a4cdaf914b01e84830639598cf8f43f2d6f18178"}
Sep 09 16:13:37.861095 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.861058    2556 scope.go:117] "RemoveContainer" containerID="a2ac0528cc0efc9271ceac53fe2a5441352b05924802b3254472faaba11d712c"
Sep 09 16:13:37.869783 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.869764    2556 scope.go:117] "RemoveContainer" containerID="a2ac0528cc0efc9271ceac53fe2a5441352b05924802b3254472faaba11d712c"
Sep 09 16:13:37.870049 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:13:37.870032    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ac0528cc0efc9271ceac53fe2a5441352b05924802b3254472faaba11d712c\": container with ID starting with a2ac0528cc0efc9271ceac53fe2a5441352b05924802b3254472faaba11d712c not found: ID does not exist" containerID="a2ac0528cc0efc9271ceac53fe2a5441352b05924802b3254472faaba11d712c"
Sep 09 16:13:37.870097 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.870057    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ac0528cc0efc9271ceac53fe2a5441352b05924802b3254472faaba11d712c"} err="failed to get container status \"a2ac0528cc0efc9271ceac53fe2a5441352b05924802b3254472faaba11d712c\": rpc error: code = NotFound desc = could not find container \"a2ac0528cc0efc9271ceac53fe2a5441352b05924802b3254472faaba11d712c\": container with ID starting with a2ac0528cc0efc9271ceac53fe2a5441352b05924802b3254472faaba11d712c not found: ID does not exist"
Sep 09 16:13:37.877153 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.877126    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"]
Sep 09 16:13:37.881642 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:37.881620    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bfb318-6b6569bcfd-j5qbx"]
Sep 09 16:13:38.067206 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.067127    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"]
Sep 09 16:13:38.067389 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.067344    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww" podUID="8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee" containerName="payload-pre-processing" containerID="cri-o://545bd91c14692503e44c400d4f05ff2894003a9a78ffdfecdf0199e097da0249" gracePeriod=30
Sep 09 16:13:38.318664 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.318602    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"
Sep 09 16:13:38.473319 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.473282    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee-plugins-config-volume\") pod \"8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee\" (UID: \"8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee\") "
Sep 09 16:13:38.473499 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.473334    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrt6s\" (UniqueName: \"kubernetes.io/projected/8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee-kube-api-access-jrt6s\") pod \"8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee\" (UID: \"8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee\") "
Sep 09 16:13:38.473724 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.473697    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee" (UID: "8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:13:38.475373 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.475349    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee-kube-api-access-jrt6s" (OuterVolumeSpecName: "kube-api-access-jrt6s") pod "8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee" (UID: "8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee"). InnerVolumeSpecName "kube-api-access-jrt6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:13:38.574310 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.574235    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrt6s\" (UniqueName: \"kubernetes.io/projected/8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee-kube-api-access-jrt6s\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:38.574310 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.574262    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:38.868295 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.868187    2556 generic.go:356] "Generic (PLEG): container finished" podID="8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee" containerID="545bd91c14692503e44c400d4f05ff2894003a9a78ffdfecdf0199e097da0249" exitCode=0
Sep 09 16:13:38.868295 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.868225    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"
Sep 09 16:13:38.868295 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.868231    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww" event={"ID":"8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee","Type":"ContainerDied","Data":"545bd91c14692503e44c400d4f05ff2894003a9a78ffdfecdf0199e097da0249"}
Sep 09 16:13:38.868295 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.868277    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww" event={"ID":"8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee","Type":"ContainerDied","Data":"7e24bf666c0c8e76f32218cd8beb847d4529201e78ace58ce3baf1b35e0062ee"}
Sep 09 16:13:38.868697 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.868302    2556 scope.go:117] "RemoveContainer" containerID="545bd91c14692503e44c400d4f05ff2894003a9a78ffdfecdf0199e097da0249"
Sep 09 16:13:38.878160 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.878143    2556 scope.go:117] "RemoveContainer" containerID="545bd91c14692503e44c400d4f05ff2894003a9a78ffdfecdf0199e097da0249"
Sep 09 16:13:38.878431 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:13:38.878409    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545bd91c14692503e44c400d4f05ff2894003a9a78ffdfecdf0199e097da0249\": container with ID starting with 545bd91c14692503e44c400d4f05ff2894003a9a78ffdfecdf0199e097da0249 not found: ID does not exist" containerID="545bd91c14692503e44c400d4f05ff2894003a9a78ffdfecdf0199e097da0249"
Sep 09 16:13:38.878532 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.878440    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545bd91c14692503e44c400d4f05ff2894003a9a78ffdfecdf0199e097da0249"} err="failed to get container status \"545bd91c14692503e44c400d4f05ff2894003a9a78ffdfecdf0199e097da0249\": rpc error: code = NotFound desc = could not find container \"545bd91c14692503e44c400d4f05ff2894003a9a78ffdfecdf0199e097da0249\": container with ID starting with 545bd91c14692503e44c400d4f05ff2894003a9a78ffdfecdf0199e097da0249 not found: ID does not exist"
Sep 09 16:13:38.889624 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.889600    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"]
Sep 09 16:13:38.895239 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:38.895217    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7b8369-68d64bc9b6-sc4ww"]
Sep 09 16:13:38.899879 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:13:38.899857    2556 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec0588b_dcf1_4cfa_93ef_0f865d8bd7ee.slice/crio-7e24bf666c0c8e76f32218cd8beb847d4529201e78ace58ce3baf1b35e0062ee\": RecentStats: unable to find data in memory cache]"
Sep 09 16:13:39.538689 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:39.538660    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:39.639589 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:39.639552    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:39.690774 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:39.690736    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm" podUID="96232610-6524-4e04-852d-886c6bdebf9e" containerName="main" containerID="cri-o://4815aad91d50eeb06e10465c4139a12cb32a00512c23eee01569dd50c35190bd" gracePeriod=45
Sep 09 16:13:39.762583 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:39.762551    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="405ef3bc-1a82-4fdf-85fe-87bee2cde743" path="/var/lib/kubelet/pods/405ef3bc-1a82-4fdf-85fe-87bee2cde743/volumes"
Sep 09 16:13:39.762918 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:39.762904    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee" path="/var/lib/kubelet/pods/8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee/volumes"
Sep 09 16:13:39.876152 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:39.876117    2556 generic.go:356] "Generic (PLEG): container finished" podID="96232610-6524-4e04-852d-886c6bdebf9e" containerID="4815aad91d50eeb06e10465c4139a12cb32a00512c23eee01569dd50c35190bd" exitCode=0
Sep 09 16:13:39.876301 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:39.876176    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm" event={"ID":"96232610-6524-4e04-852d-886c6bdebf9e","Type":"ContainerDied","Data":"4815aad91d50eeb06e10465c4139a12cb32a00512c23eee01569dd50c35190bd"}
Sep 09 16:13:39.929912 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:39.929886    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:13:40.084608 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.084510    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96232610-6524-4e04-852d-886c6bdebf9e-tls-certs\") pod \"96232610-6524-4e04-852d-886c6bdebf9e\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") "
Sep 09 16:13:40.084608 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.084583    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-model-cache\") pod \"96232610-6524-4e04-852d-886c6bdebf9e\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") "
Sep 09 16:13:40.084842 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.084619    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-tmp-dir\") pod \"96232610-6524-4e04-852d-886c6bdebf9e\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") "
Sep 09 16:13:40.084842 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.084653    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-dshm\") pod \"96232610-6524-4e04-852d-886c6bdebf9e\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") "
Sep 09 16:13:40.084842 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.084726    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvbq5\" (UniqueName: \"kubernetes.io/projected/96232610-6524-4e04-852d-886c6bdebf9e-kube-api-access-kvbq5\") pod \"96232610-6524-4e04-852d-886c6bdebf9e\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") "
Sep 09 16:13:40.084842 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.084755    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-home\") pod \"96232610-6524-4e04-852d-886c6bdebf9e\" (UID: \"96232610-6524-4e04-852d-886c6bdebf9e\") "
Sep 09 16:13:40.085042 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.084844    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "96232610-6524-4e04-852d-886c6bdebf9e" (UID: "96232610-6524-4e04-852d-886c6bdebf9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:13:40.085042 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.084942    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96232610-6524-4e04-852d-886c6bdebf9e" (UID: "96232610-6524-4e04-852d-886c6bdebf9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:13:40.085159 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.085083    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-tmp-dir\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:40.085159 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.085104    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-model-cache\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:40.085159 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.085109    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-home" (OuterVolumeSpecName: "home") pod "96232610-6524-4e04-852d-886c6bdebf9e" (UID: "96232610-6524-4e04-852d-886c6bdebf9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:13:40.086894 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.086863    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-dshm" (OuterVolumeSpecName: "dshm") pod "96232610-6524-4e04-852d-886c6bdebf9e" (UID: "96232610-6524-4e04-852d-886c6bdebf9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:13:40.086894 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.086883    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96232610-6524-4e04-852d-886c6bdebf9e-kube-api-access-kvbq5" (OuterVolumeSpecName: "kube-api-access-kvbq5") pod "96232610-6524-4e04-852d-886c6bdebf9e" (UID: "96232610-6524-4e04-852d-886c6bdebf9e"). InnerVolumeSpecName "kube-api-access-kvbq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:13:40.087050 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.086906    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96232610-6524-4e04-852d-886c6bdebf9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96232610-6524-4e04-852d-886c6bdebf9e" (UID: "96232610-6524-4e04-852d-886c6bdebf9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:13:40.185506 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.185458    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-dshm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:40.185506 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.185502    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvbq5\" (UniqueName: \"kubernetes.io/projected/96232610-6524-4e04-852d-886c6bdebf9e-kube-api-access-kvbq5\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:40.185506 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.185511    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96232610-6524-4e04-852d-886c6bdebf9e-home\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:40.185732 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.185521    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96232610-6524-4e04-852d-886c6bdebf9e-tls-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:40.882674 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.882642    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"
Sep 09 16:13:40.882674 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.882632    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm" event={"ID":"96232610-6524-4e04-852d-886c6bdebf9e","Type":"ContainerDied","Data":"4488310d243d34cc1808e53a7046ee9859d12ed56d250c871f2f179404b79420"}
Sep 09 16:13:40.883174 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.882703    2556 scope.go:117] "RemoveContainer" containerID="4815aad91d50eeb06e10465c4139a12cb32a00512c23eee01569dd50c35190bd"
Sep 09 16:13:40.908667 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.908635    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"]
Sep 09 16:13:40.911144 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:40.911119    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7b8369/test-model-7b8369-kserve-7c69d94547-kxdvm"]
Sep 09 16:13:41.763802 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:41.763767    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96232610-6524-4e04-852d-886c6bdebf9e" path="/var/lib/kubelet/pods/96232610-6524-4e04-852d-886c6bdebf9e/volumes"
Sep 09 16:13:42.055962 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.055870    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9" podUID="75e9310c-aa7c-4894-a818-0d4c0adee060" containerName="main" containerID="cri-o://a804ca0696eab6aa0e9dac020d0e0d9c72d5abb1ff7db459c4baf3522932f167" gracePeriod=45
Sep 09 16:13:42.298846 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.298820    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:13:42.404374 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.404275    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-home\") pod \"75e9310c-aa7c-4894-a818-0d4c0adee060\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") "
Sep 09 16:13:42.404374 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.404359    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlzjw\" (UniqueName: \"kubernetes.io/projected/75e9310c-aa7c-4894-a818-0d4c0adee060-kube-api-access-xlzjw\") pod \"75e9310c-aa7c-4894-a818-0d4c0adee060\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") "
Sep 09 16:13:42.404647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.404430    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-tmp-dir\") pod \"75e9310c-aa7c-4894-a818-0d4c0adee060\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") "
Sep 09 16:13:42.404647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.404571    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-home" (OuterVolumeSpecName: "home") pod "75e9310c-aa7c-4894-a818-0d4c0adee060" (UID: "75e9310c-aa7c-4894-a818-0d4c0adee060"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:13:42.404647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.404590    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-model-cache\") pod \"75e9310c-aa7c-4894-a818-0d4c0adee060\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") "
Sep 09 16:13:42.404647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.404637    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e9310c-aa7c-4894-a818-0d4c0adee060-tls-certs\") pod \"75e9310c-aa7c-4894-a818-0d4c0adee060\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") "
Sep 09 16:13:42.404837 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.404672    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-dshm\") pod \"75e9310c-aa7c-4894-a818-0d4c0adee060\" (UID: \"75e9310c-aa7c-4894-a818-0d4c0adee060\") "
Sep 09 16:13:42.404837 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.404668    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75e9310c-aa7c-4894-a818-0d4c0adee060" (UID: "75e9310c-aa7c-4894-a818-0d4c0adee060"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:13:42.404837 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.404807    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-model-cache" (OuterVolumeSpecName: "model-cache") pod "75e9310c-aa7c-4894-a818-0d4c0adee060" (UID: "75e9310c-aa7c-4894-a818-0d4c0adee060"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:13:42.405011 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.404996    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-home\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:42.405049 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.405018    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-tmp-dir\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:42.405049 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.405033    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-model-cache\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:42.406738 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.406710    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-dshm" (OuterVolumeSpecName: "dshm") pod "75e9310c-aa7c-4894-a818-0d4c0adee060" (UID: "75e9310c-aa7c-4894-a818-0d4c0adee060"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:13:42.406738 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.406732    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e9310c-aa7c-4894-a818-0d4c0adee060-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75e9310c-aa7c-4894-a818-0d4c0adee060" (UID: "75e9310c-aa7c-4894-a818-0d4c0adee060"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:13:42.406891 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.406774    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e9310c-aa7c-4894-a818-0d4c0adee060-kube-api-access-xlzjw" (OuterVolumeSpecName: "kube-api-access-xlzjw") pod "75e9310c-aa7c-4894-a818-0d4c0adee060" (UID: "75e9310c-aa7c-4894-a818-0d4c0adee060"). InnerVolumeSpecName "kube-api-access-xlzjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:13:42.505850 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.505821    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlzjw\" (UniqueName: \"kubernetes.io/projected/75e9310c-aa7c-4894-a818-0d4c0adee060-kube-api-access-xlzjw\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:42.505850 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.505845    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e9310c-aa7c-4894-a818-0d4c0adee060-tls-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:42.505850 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.505855    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e9310c-aa7c-4894-a818-0d4c0adee060-dshm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:42.836604 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.836573    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"
Sep 09 16:13:42.836808 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.836644    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"
Sep 09 16:13:42.895766 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.895732    2556 generic.go:356] "Generic (PLEG): container finished" podID="75e9310c-aa7c-4894-a818-0d4c0adee060" containerID="a804ca0696eab6aa0e9dac020d0e0d9c72d5abb1ff7db459c4baf3522932f167" exitCode=0
Sep 09 16:13:42.895904 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.895774    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"
Sep 09 16:13:42.895904 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.895767    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9" event={"ID":"75e9310c-aa7c-4894-a818-0d4c0adee060","Type":"ContainerDied","Data":"a804ca0696eab6aa0e9dac020d0e0d9c72d5abb1ff7db459c4baf3522932f167"}
Sep 09 16:13:42.895904 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.895827    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9" event={"ID":"75e9310c-aa7c-4894-a818-0d4c0adee060","Type":"ContainerDied","Data":"4fe0a1b82e8df9df77fd069e2cf31d5967d2e99592b13f887182648e481d1164"}
Sep 09 16:13:42.895904 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.895853    2556 scope.go:117] "RemoveContainer" containerID="a804ca0696eab6aa0e9dac020d0e0d9c72d5abb1ff7db459c4baf3522932f167"
Sep 09 16:13:42.905990 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.905970    2556 scope.go:117] "RemoveContainer" containerID="a804ca0696eab6aa0e9dac020d0e0d9c72d5abb1ff7db459c4baf3522932f167"
Sep 09 16:13:42.906264 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:13:42.906243    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a804ca0696eab6aa0e9dac020d0e0d9c72d5abb1ff7db459c4baf3522932f167\": container with ID starting with a804ca0696eab6aa0e9dac020d0e0d9c72d5abb1ff7db459c4baf3522932f167 not found: ID does not exist" containerID="a804ca0696eab6aa0e9dac020d0e0d9c72d5abb1ff7db459c4baf3522932f167"
Sep 09 16:13:42.906338 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.906274    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a804ca0696eab6aa0e9dac020d0e0d9c72d5abb1ff7db459c4baf3522932f167"} err="failed to get container status \"a804ca0696eab6aa0e9dac020d0e0d9c72d5abb1ff7db459c4baf3522932f167\": rpc error: code = NotFound desc = could not find container \"a804ca0696eab6aa0e9dac020d0e0d9c72d5abb1ff7db459c4baf3522932f167\": container with ID starting with a804ca0696eab6aa0e9dac020d0e0d9c72d5abb1ff7db459c4baf3522932f167 not found: ID does not exist"
Sep 09 16:13:42.917007 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.916982    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"]
Sep 09 16:13:42.920348 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:42.920316    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c23ff1/test-model-c23ff1-kserve-5f9bb84f87-mwtq9"]
Sep 09 16:13:43.564184 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:43.564135    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:43.665675 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:43.665641    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:43.765283 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:43.765248    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75e9310c-aa7c-4894-a818-0d4c0adee060" path="/var/lib/kubelet/pods/75e9310c-aa7c-4894-a818-0d4c0adee060/volumes"
Sep 09 16:13:43.828596 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:43.828458    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:44.324143 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:44.324101    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:44.588067 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:44.587966    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:44.773626 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:44.773587    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:45.179657 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.179624    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:45.279123 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.279092    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:45.431741 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.431653    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:45.769402 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.769322    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q"]
Sep 09 16:13:45.769880 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.769787    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="405ef3bc-1a82-4fdf-85fe-87bee2cde743" containerName="payload-pre-processing"
Sep 09 16:13:45.769880 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.769806    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="405ef3bc-1a82-4fdf-85fe-87bee2cde743" containerName="payload-pre-processing"
Sep 09 16:13:45.769880 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.769818    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96232610-6524-4e04-852d-886c6bdebf9e" containerName="main"
Sep 09 16:13:45.769880 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.769826    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="96232610-6524-4e04-852d-886c6bdebf9e" containerName="main"
Sep 09 16:13:45.769880 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.769844    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee" containerName="payload-pre-processing"
Sep 09 16:13:45.769880 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.769853    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee" containerName="payload-pre-processing"
Sep 09 16:13:45.769880 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.769867    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e9310c-aa7c-4894-a818-0d4c0adee060" containerName="main"
Sep 09 16:13:45.769880 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.769874    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e9310c-aa7c-4894-a818-0d4c0adee060" containerName="main"
Sep 09 16:13:45.770283 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.769887    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ad114a-db9b-4b2e-93a6-54d580c9fea9" containerName="revoke-keys"
Sep 09 16:13:45.770283 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.769895    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ad114a-db9b-4b2e-93a6-54d580c9fea9" containerName="revoke-keys"
Sep 09 16:13:45.770283 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.769985    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec0588b-dcf1-4cfa-93ef-0f865d8bd7ee" containerName="payload-pre-processing"
Sep 09 16:13:45.770283 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.769996    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="12ad114a-db9b-4b2e-93a6-54d580c9fea9" containerName="revoke-keys"
Sep 09 16:13:45.770283 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.770007    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="96232610-6524-4e04-852d-886c6bdebf9e" containerName="main"
Sep 09 16:13:45.770283 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.770019    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e9310c-aa7c-4894-a818-0d4c0adee060" containerName="main"
Sep 09 16:13:45.770283 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.770027    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="405ef3bc-1a82-4fdf-85fe-87bee2cde743" containerName="payload-pre-processing"
Sep 09 16:13:45.774303 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.774274    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q"]
Sep 09 16:13:45.774449 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.774376    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q"
Sep 09 16:13:45.776674 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.776653    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:13:45.776783 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.776673    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\""
Sep 09 16:13:45.876421 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.876382    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:45.934205 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:45.934166    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff67b802-5a69-470c-985e-655a4a89ae0c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q\" (UID: \"ff67b802-5a69-470c-985e-655a4a89ae0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q"
Sep 09 16:13:46.035203 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:46.035108    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff67b802-5a69-470c-985e-655a4a89ae0c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q\" (UID: \"ff67b802-5a69-470c-985e-655a4a89ae0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q"
Sep 09 16:13:46.035790 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:46.035770    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff67b802-5a69-470c-985e-655a4a89ae0c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q\" (UID: \"ff67b802-5a69-470c-985e-655a4a89ae0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q"
Sep 09 16:13:46.083742 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:46.083706    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q"
Sep 09 16:13:46.352313 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:46.352285    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q"]
Sep 09 16:13:46.354304 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:13:46.354265    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff67b802_5a69_470c_985e_655a4a89ae0c.slice/crio-23c3bd7edf30a003b8eaf196f4f02c08f962accbf0d8aa32c3199d49595ccdb4 WatchSource:0}: Error finding container 23c3bd7edf30a003b8eaf196f4f02c08f962accbf0d8aa32c3199d49595ccdb4: Status 404 returned error can't find the container with id 23c3bd7edf30a003b8eaf196f4f02c08f962accbf0d8aa32c3199d49595ccdb4
Sep 09 16:13:46.915756 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:46.915722    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q" event={"ID":"ff67b802-5a69-470c-985e-655a4a89ae0c","Type":"ContainerStarted","Data":"8689fa65dc67b3e4d7f5144367b9b17bd3d9c82387a4eac622fdabd9094710e4"}
Sep 09 16:13:46.916129 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:46.915773    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q" event={"ID":"ff67b802-5a69-470c-985e-655a4a89ae0c","Type":"ContainerStarted","Data":"23c3bd7edf30a003b8eaf196f4f02c08f962accbf0d8aa32c3199d49595ccdb4"}
Sep 09 16:13:46.929486 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:46.929112    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q" podStartSLOduration=1.9290914479999999 podStartE2EDuration="1.929091448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:13:46.928858499 +0000 UTC m=+1331.795369720" watchObservedRunningTime="2026-09-09 16:13:46.929091448 +0000 UTC m=+1331.795602689"
Sep 09 16:13:47.922287 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:47.922253    2556 generic.go:356] "Generic (PLEG): container finished" podID="ff67b802-5a69-470c-985e-655a4a89ae0c" containerID="8689fa65dc67b3e4d7f5144367b9b17bd3d9c82387a4eac622fdabd9094710e4" exitCode=0
Sep 09 16:13:47.922781 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:47.922321    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q" event={"ID":"ff67b802-5a69-470c-985e-655a4a89ae0c","Type":"ContainerDied","Data":"8689fa65dc67b3e4d7f5144367b9b17bd3d9c82387a4eac622fdabd9094710e4"}
Sep 09 16:13:49.042997 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:49.042956    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:49.075604 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:49.075581    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q"
Sep 09 16:13:49.115245 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:49.115214    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:49.253247 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:49.253162    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:49.262054 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:49.262036    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff67b802-5a69-470c-985e-655a4a89ae0c-maas-api-service-ca\") pod \"ff67b802-5a69-470c-985e-655a4a89ae0c\" (UID: \"ff67b802-5a69-470c-985e-655a4a89ae0c\") "
Sep 09 16:13:49.262339 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:49.262316    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff67b802-5a69-470c-985e-655a4a89ae0c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff67b802-5a69-470c-985e-655a4a89ae0c" (UID: "ff67b802-5a69-470c-985e-655a4a89ae0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:13:49.363580 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:49.363547    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff67b802-5a69-470c-985e-655a4a89ae0c-maas-api-service-ca\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:49.934426 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:49.934383    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q"
Sep 09 16:13:49.934618 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:49.934383    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q" event={"ID":"ff67b802-5a69-470c-985e-655a4a89ae0c","Type":"ContainerDied","Data":"23c3bd7edf30a003b8eaf196f4f02c08f962accbf0d8aa32c3199d49595ccdb4"}
Sep 09 16:13:49.934618 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:49.934548    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23c3bd7edf30a003b8eaf196f4f02c08f962accbf0d8aa32c3199d49595ccdb4"
Sep 09 16:13:50.855174 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:50.855130    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:52.839948 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.839912    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"]
Sep 09 16:13:52.840397 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.840382    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff67b802-5a69-470c-985e-655a4a89ae0c" containerName="revoke-keys"
Sep 09 16:13:52.840445 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.840401    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff67b802-5a69-470c-985e-655a4a89ae0c" containerName="revoke-keys"
Sep 09 16:13:52.840534 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.840521    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff67b802-5a69-470c-985e-655a4a89ae0c" containerName="revoke-keys"
Sep 09 16:13:52.843727 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.843701    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.846805 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.846782    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-a372ad\"/\"openshift-service-ca.crt\""
Sep 09 16:13:52.846936 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.846804    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-a372ad\"/\"default-dockercfg-tgdx6\""
Sep 09 16:13:52.846936 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.846880    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-a372ad\"/\"noten-model-a372ad-kserve-self-signed-certs\""
Sep 09 16:13:52.847041 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.847029    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-a372ad\"/\"kube-root-ca.crt\""
Sep 09 16:13:52.851371 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.851249    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"]
Sep 09 16:13:52.869825 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.869782    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:52.891989 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.891961    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8j4b\" (UniqueName: \"kubernetes.io/projected/4d9b272d-1123-455c-944b-bcc6f12710d9-kube-api-access-n8j4b\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.892086 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.891994    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-home\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.892086 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.892013    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-model-cache\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.892086 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.892055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-dshm\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.892086 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.892082    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9b272d-1123-455c-944b-bcc6f12710d9-tls-certs\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.892216 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.892157    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-tmp-dir\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.992815 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.992779    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9b272d-1123-455c-944b-bcc6f12710d9-tls-certs\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.993011 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.992845    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-tmp-dir\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.993011 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.992891    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8j4b\" (UniqueName: \"kubernetes.io/projected/4d9b272d-1123-455c-944b-bcc6f12710d9-kube-api-access-n8j4b\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.993011 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.992930    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-home\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.993011 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.992956    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-model-cache\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.993011 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.992990    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-dshm\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.993338 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.993308    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-tmp-dir\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.993485 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.993319    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-home\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.993485 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.993377    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-model-cache\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.995115 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.995084    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-dshm\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:52.995280 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:52.995262    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9b272d-1123-455c-944b-bcc6f12710d9-tls-certs\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:53.000268 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:53.000246    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8j4b\" (UniqueName: \"kubernetes.io/projected/4d9b272d-1123-455c-944b-bcc6f12710d9-kube-api-access-n8j4b\") pod \"noten-model-a372ad-kserve-7d6d6c4cdc-kbljp\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") " pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:53.154422 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:53.154333    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:13:53.423612 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:53.423586    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"]
Sep 09 16:13:53.425774 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:13:53.425745    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d9b272d_1123_455c_944b_bcc6f12710d9.slice/crio-b8b722343cc7a00b0d68fc671ce7af62110dabbc48d735267643ad57fca19adb WatchSource:0}: Error finding container b8b722343cc7a00b0d68fc671ce7af62110dabbc48d735267643ad57fca19adb: Status 404 returned error can't find the container with id b8b722343cc7a00b0d68fc671ce7af62110dabbc48d735267643ad57fca19adb
Sep 09 16:13:53.513417 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:53.513379    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:53.952901 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:53.952866    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp" event={"ID":"4d9b272d-1123-455c-944b-bcc6f12710d9","Type":"ContainerStarted","Data":"028fa1a71b2f9a69c62f5af85e7a3f3dc977c922f52a7a5280b835e2c17d93bc"}
Sep 09 16:13:53.953302 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:53.952916    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp" event={"ID":"4d9b272d-1123-455c-944b-bcc6f12710d9","Type":"ContainerStarted","Data":"b8b722343cc7a00b0d68fc671ce7af62110dabbc48d735267643ad57fca19adb"}
Sep 09 16:13:53.968523 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:53.968455    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp" podStartSLOduration=1.968440303 podStartE2EDuration="1.968440303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:13:53.967256361 +0000 UTC m=+1338.833767580" watchObservedRunningTime="2026-09-09 16:13:53.968440303 +0000 UTC m=+1338.834951524"
Sep 09 16:13:54.176099 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:54.176044    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:54.270323 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:54.270233    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:54.583312 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:54.583221    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:55.913643 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:55.913608    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:56.501314 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.501280    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:56.526326 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.526293    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"]
Sep 09 16:13:56.526557 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.526531    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48" podUID="06afc164-5e10-4dba-b5fe-bc3d6fc921d5" containerName="payload-processing" containerID="cri-o://6e40ee000a89000ccda417d53f75aa0deab56336a3c5a30c6711896760abcdaf" gracePeriod=30
Sep 09 16:13:56.533154 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.533127    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"]
Sep 09 16:13:56.533434 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.533333    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj" podUID="3ce4a718-2070-4af5-be6f-a498a7a38af5" containerName="payload-pre-processing" containerID="cri-o://e3b854eca4ae49ebbbd1ffa73e3183f7643d9665cf7ac2c4ee41f0fd25e91a7b" gracePeriod=30
Sep 09 16:13:56.573165 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.573136    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:56.674038 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.673959    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:56.689042 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.688993    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q"]
Sep 09 16:13:56.691340 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.691313    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c23ff1-jzp4q"]
Sep 09 16:13:56.744621 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.744589    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:56.800650 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.800630    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"
Sep 09 16:13:56.803349 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.803329    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"
Sep 09 16:13:56.823807 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.823784    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlxlm\" (UniqueName: \"kubernetes.io/projected/06afc164-5e10-4dba-b5fe-bc3d6fc921d5-kube-api-access-tlxlm\") pod \"06afc164-5e10-4dba-b5fe-bc3d6fc921d5\" (UID: \"06afc164-5e10-4dba-b5fe-bc3d6fc921d5\") "
Sep 09 16:13:56.823930 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.823819    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8677\" (UniqueName: \"kubernetes.io/projected/3ce4a718-2070-4af5-be6f-a498a7a38af5-kube-api-access-z8677\") pod \"3ce4a718-2070-4af5-be6f-a498a7a38af5\" (UID: \"3ce4a718-2070-4af5-be6f-a498a7a38af5\") "
Sep 09 16:13:56.823930 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.823865    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06afc164-5e10-4dba-b5fe-bc3d6fc921d5-plugins-config-volume\") pod \"06afc164-5e10-4dba-b5fe-bc3d6fc921d5\" (UID: \"06afc164-5e10-4dba-b5fe-bc3d6fc921d5\") "
Sep 09 16:13:56.823930 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.823913    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce4a718-2070-4af5-be6f-a498a7a38af5-plugins-config-volume\") pod \"3ce4a718-2070-4af5-be6f-a498a7a38af5\" (UID: \"3ce4a718-2070-4af5-be6f-a498a7a38af5\") "
Sep 09 16:13:56.824301 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.824275    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06afc164-5e10-4dba-b5fe-bc3d6fc921d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "06afc164-5e10-4dba-b5fe-bc3d6fc921d5" (UID: "06afc164-5e10-4dba-b5fe-bc3d6fc921d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:13:56.824431 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.824402    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce4a718-2070-4af5-be6f-a498a7a38af5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ce4a718-2070-4af5-be6f-a498a7a38af5" (UID: "3ce4a718-2070-4af5-be6f-a498a7a38af5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:13:56.825901 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.825878    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce4a718-2070-4af5-be6f-a498a7a38af5-kube-api-access-z8677" (OuterVolumeSpecName: "kube-api-access-z8677") pod "3ce4a718-2070-4af5-be6f-a498a7a38af5" (UID: "3ce4a718-2070-4af5-be6f-a498a7a38af5"). InnerVolumeSpecName "kube-api-access-z8677". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:13:56.825983 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.825921    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06afc164-5e10-4dba-b5fe-bc3d6fc921d5-kube-api-access-tlxlm" (OuterVolumeSpecName: "kube-api-access-tlxlm") pod "06afc164-5e10-4dba-b5fe-bc3d6fc921d5" (UID: "06afc164-5e10-4dba-b5fe-bc3d6fc921d5"). InnerVolumeSpecName "kube-api-access-tlxlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:13:56.925450 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.925418    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06afc164-5e10-4dba-b5fe-bc3d6fc921d5-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:56.925450 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.925447    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce4a718-2070-4af5-be6f-a498a7a38af5-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:56.925866 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.925460    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlxlm\" (UniqueName: \"kubernetes.io/projected/06afc164-5e10-4dba-b5fe-bc3d6fc921d5-kube-api-access-tlxlm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:56.925866 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.925498    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8677\" (UniqueName: \"kubernetes.io/projected/3ce4a718-2070-4af5-be6f-a498a7a38af5-kube-api-access-z8677\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:13:56.967240 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.967202    2556 generic.go:356] "Generic (PLEG): container finished" podID="06afc164-5e10-4dba-b5fe-bc3d6fc921d5" containerID="6e40ee000a89000ccda417d53f75aa0deab56336a3c5a30c6711896760abcdaf" exitCode=0
Sep 09 16:13:56.967397 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.967245    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"
Sep 09 16:13:56.967397 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.967237    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48" event={"ID":"06afc164-5e10-4dba-b5fe-bc3d6fc921d5","Type":"ContainerDied","Data":"6e40ee000a89000ccda417d53f75aa0deab56336a3c5a30c6711896760abcdaf"}
Sep 09 16:13:56.967545 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.967434    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48" event={"ID":"06afc164-5e10-4dba-b5fe-bc3d6fc921d5","Type":"ContainerDied","Data":"aeb0935a48072f8944aa65c703c67af8fa15aa18219f22c7c678a107116e02ad"}
Sep 09 16:13:56.967545 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.967482    2556 scope.go:117] "RemoveContainer" containerID="6e40ee000a89000ccda417d53f75aa0deab56336a3c5a30c6711896760abcdaf"
Sep 09 16:13:56.968766 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.968742    2556 generic.go:356] "Generic (PLEG): container finished" podID="3ce4a718-2070-4af5-be6f-a498a7a38af5" containerID="e3b854eca4ae49ebbbd1ffa73e3183f7643d9665cf7ac2c4ee41f0fd25e91a7b" exitCode=0
Sep 09 16:13:56.968870 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.968781    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"
Sep 09 16:13:56.968870 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.968794    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj" event={"ID":"3ce4a718-2070-4af5-be6f-a498a7a38af5","Type":"ContainerDied","Data":"e3b854eca4ae49ebbbd1ffa73e3183f7643d9665cf7ac2c4ee41f0fd25e91a7b"}
Sep 09 16:13:56.968870 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.968832    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj" event={"ID":"3ce4a718-2070-4af5-be6f-a498a7a38af5","Type":"ContainerDied","Data":"191aa8d900a2b2efdc0e5b9e48b243622597c373c23ff1520f315a8fa219b92d"}
Sep 09 16:13:56.976161 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.976141    2556 scope.go:117] "RemoveContainer" containerID="6e40ee000a89000ccda417d53f75aa0deab56336a3c5a30c6711896760abcdaf"
Sep 09 16:13:56.976508 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:13:56.976488    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e40ee000a89000ccda417d53f75aa0deab56336a3c5a30c6711896760abcdaf\": container with ID starting with 6e40ee000a89000ccda417d53f75aa0deab56336a3c5a30c6711896760abcdaf not found: ID does not exist" containerID="6e40ee000a89000ccda417d53f75aa0deab56336a3c5a30c6711896760abcdaf"
Sep 09 16:13:56.976602 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.976514    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e40ee000a89000ccda417d53f75aa0deab56336a3c5a30c6711896760abcdaf"} err="failed to get container status \"6e40ee000a89000ccda417d53f75aa0deab56336a3c5a30c6711896760abcdaf\": rpc error: code = NotFound desc = could not find container \"6e40ee000a89000ccda417d53f75aa0deab56336a3c5a30c6711896760abcdaf\": container with ID starting with 6e40ee000a89000ccda417d53f75aa0deab56336a3c5a30c6711896760abcdaf not found: ID does not exist"
Sep 09 16:13:56.976602 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.976529    2556 scope.go:117] "RemoveContainer" containerID="e3b854eca4ae49ebbbd1ffa73e3183f7643d9665cf7ac2c4ee41f0fd25e91a7b"
Sep 09 16:13:56.984596 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.984577    2556 scope.go:117] "RemoveContainer" containerID="e3b854eca4ae49ebbbd1ffa73e3183f7643d9665cf7ac2c4ee41f0fd25e91a7b"
Sep 09 16:13:56.984854 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:13:56.984833    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b854eca4ae49ebbbd1ffa73e3183f7643d9665cf7ac2c4ee41f0fd25e91a7b\": container with ID starting with e3b854eca4ae49ebbbd1ffa73e3183f7643d9665cf7ac2c4ee41f0fd25e91a7b not found: ID does not exist" containerID="e3b854eca4ae49ebbbd1ffa73e3183f7643d9665cf7ac2c4ee41f0fd25e91a7b"
Sep 09 16:13:56.984927 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.984867    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b854eca4ae49ebbbd1ffa73e3183f7643d9665cf7ac2c4ee41f0fd25e91a7b"} err="failed to get container status \"e3b854eca4ae49ebbbd1ffa73e3183f7643d9665cf7ac2c4ee41f0fd25e91a7b\": rpc error: code = NotFound desc = could not find container \"e3b854eca4ae49ebbbd1ffa73e3183f7643d9665cf7ac2c4ee41f0fd25e91a7b\": container with ID starting with e3b854eca4ae49ebbbd1ffa73e3183f7643d9665cf7ac2c4ee41f0fd25e91a7b not found: ID does not exist"
Sep 09 16:13:56.992387 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.992363    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"]
Sep 09 16:13:56.996527 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:56.996505    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c23ff1-57c5b7fb9-vvm48"]
Sep 09 16:13:57.005387 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:57.005366    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"]
Sep 09 16:13:57.007674 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:57.007654    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c23ff1-664f469d6b-w7fdj"]
Sep 09 16:13:57.053954 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:57.053925    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:13:57.762644 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:57.762573    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06afc164-5e10-4dba-b5fe-bc3d6fc921d5" path="/var/lib/kubelet/pods/06afc164-5e10-4dba-b5fe-bc3d6fc921d5/volumes"
Sep 09 16:13:57.762934 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:57.762921    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ce4a718-2070-4af5-be6f-a498a7a38af5" path="/var/lib/kubelet/pods/3ce4a718-2070-4af5-be6f-a498a7a38af5/volumes"
Sep 09 16:13:57.763206 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:13:57.763194    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff67b802-5a69-470c-985e-655a4a89ae0c" path="/var/lib/kubelet/pods/ff67b802-5a69-470c-985e-655a4a89ae0c/volumes"
Sep 09 16:14:00.611608 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:00.611572    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:00.712439 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:00.712402    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:01.063436 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:01.063403    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:02.836882 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:02.836845    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:03.154703 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:03.154602    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:14:03.154703 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:03.154652    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:14:03.167373 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:03.167341    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:14:04.012672 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:04.012635    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:14:06.377260 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:06.377220    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:06.432549 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:06.432516    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:06.868041 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:06.868010    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:06.967233 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:06.967186    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:08.072201 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.072159    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:08.174582 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.174549    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:08.272607 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.272571    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:08.541452 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.541412    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:08.644883 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.644849    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:08.678685 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.678652    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"]
Sep 09 16:14:08.679049 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.679030    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06afc164-5e10-4dba-b5fe-bc3d6fc921d5" containerName="payload-processing"
Sep 09 16:14:08.679135 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.679052    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="06afc164-5e10-4dba-b5fe-bc3d6fc921d5" containerName="payload-processing"
Sep 09 16:14:08.679135 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.679079    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce4a718-2070-4af5-be6f-a498a7a38af5" containerName="payload-pre-processing"
Sep 09 16:14:08.679135 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.679088    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce4a718-2070-4af5-be6f-a498a7a38af5" containerName="payload-pre-processing"
Sep 09 16:14:08.679286 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.679173    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="06afc164-5e10-4dba-b5fe-bc3d6fc921d5" containerName="payload-processing"
Sep 09 16:14:08.679286 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.679187    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ce4a718-2070-4af5-be6f-a498a7a38af5" containerName="payload-pre-processing"
Sep 09 16:14:08.681126 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.681105    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"
Sep 09 16:14:08.683808 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.683786    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-131689\""
Sep 09 16:14:08.699653 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.699625    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"]
Sep 09 16:14:08.718215 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.718183    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"]
Sep 09 16:14:08.720687 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.720660    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"
Sep 09 16:14:08.724588 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.724564    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-131689-dockercfg-ptq9n\""
Sep 09 16:14:08.732841 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.732816    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a904a14-212b-42ae-9b1f-ab0ef31a4bac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn\" (UID: \"2a904a14-212b-42ae-9b1f-ab0ef31a4bac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"
Sep 09 16:14:08.732934 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.732894    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtfpb\" (UniqueName: \"kubernetes.io/projected/2a904a14-212b-42ae-9b1f-ab0ef31a4bac-kube-api-access-vtfpb\") pod \"payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn\" (UID: \"2a904a14-212b-42ae-9b1f-ab0ef31a4bac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"
Sep 09 16:14:08.739427 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.739406    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"]
Sep 09 16:14:08.834335 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.834259    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtfpb\" (UniqueName: \"kubernetes.io/projected/2a904a14-212b-42ae-9b1f-ab0ef31a4bac-kube-api-access-vtfpb\") pod \"payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn\" (UID: \"2a904a14-212b-42ae-9b1f-ab0ef31a4bac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"
Sep 09 16:14:08.834335 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.834317    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c843ebc4-ceb1-43a6-a30a-136f5a93140b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6\" (UID: \"c843ebc4-ceb1-43a6-a30a-136f5a93140b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"
Sep 09 16:14:08.834531 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.834340    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a904a14-212b-42ae-9b1f-ab0ef31a4bac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn\" (UID: \"2a904a14-212b-42ae-9b1f-ab0ef31a4bac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"
Sep 09 16:14:08.834531 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.834370    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpt7n\" (UniqueName: \"kubernetes.io/projected/c843ebc4-ceb1-43a6-a30a-136f5a93140b-kube-api-access-fpt7n\") pod \"payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6\" (UID: \"c843ebc4-ceb1-43a6-a30a-136f5a93140b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"
Sep 09 16:14:08.835065 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.835047    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a904a14-212b-42ae-9b1f-ab0ef31a4bac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn\" (UID: \"2a904a14-212b-42ae-9b1f-ab0ef31a4bac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"
Sep 09 16:14:08.846973 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.846951    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtfpb\" (UniqueName: \"kubernetes.io/projected/2a904a14-212b-42ae-9b1f-ab0ef31a4bac-kube-api-access-vtfpb\") pod \"payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn\" (UID: \"2a904a14-212b-42ae-9b1f-ab0ef31a4bac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"
Sep 09 16:14:08.935308 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.935271    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c843ebc4-ceb1-43a6-a30a-136f5a93140b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6\" (UID: \"c843ebc4-ceb1-43a6-a30a-136f5a93140b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"
Sep 09 16:14:08.935492 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.935337    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpt7n\" (UniqueName: \"kubernetes.io/projected/c843ebc4-ceb1-43a6-a30a-136f5a93140b-kube-api-access-fpt7n\") pod \"payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6\" (UID: \"c843ebc4-ceb1-43a6-a30a-136f5a93140b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"
Sep 09 16:14:08.936050 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.936030    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c843ebc4-ceb1-43a6-a30a-136f5a93140b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6\" (UID: \"c843ebc4-ceb1-43a6-a30a-136f5a93140b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"
Sep 09 16:14:08.943126 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.943103    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpt7n\" (UniqueName: \"kubernetes.io/projected/c843ebc4-ceb1-43a6-a30a-136f5a93140b-kube-api-access-fpt7n\") pod \"payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6\" (UID: \"c843ebc4-ceb1-43a6-a30a-136f5a93140b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"
Sep 09 16:14:08.991234 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:08.991203    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"
Sep 09 16:14:09.030888 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:09.030856    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"
Sep 09 16:14:09.263265 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:09.263198    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"]
Sep 09 16:14:09.265433 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:14:09.265406    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a904a14_212b_42ae_9b1f_ab0ef31a4bac.slice/crio-f29872b8942d8eb866f1d55db59748d9dc0362a98b4cd2b76a4f6e1446870a16 WatchSource:0}: Error finding container f29872b8942d8eb866f1d55db59748d9dc0362a98b4cd2b76a4f6e1446870a16: Status 404 returned error can't find the container with id f29872b8942d8eb866f1d55db59748d9dc0362a98b4cd2b76a4f6e1446870a16
Sep 09 16:14:09.312363 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:09.312341    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"]
Sep 09 16:14:09.314779 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:14:09.314751    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc843ebc4_ceb1_43a6_a30a_136f5a93140b.slice/crio-9af44b17412217a04314617772d3f6e416ba6bf8edca9e933651fc61a6aeced0 WatchSource:0}: Error finding container 9af44b17412217a04314617772d3f6e416ba6bf8edca9e933651fc61a6aeced0: Status 404 returned error can't find the container with id 9af44b17412217a04314617772d3f6e416ba6bf8edca9e933651fc61a6aeced0
Sep 09 16:14:10.033337 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:10.033300    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6" event={"ID":"c843ebc4-ceb1-43a6-a30a-136f5a93140b","Type":"ContainerStarted","Data":"97943b839a74eceb069de6260cf7da32ac9a4014e00d5d2feaf7c50d51eddd71"}
Sep 09 16:14:10.033559 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:10.033357    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6" event={"ID":"c843ebc4-ceb1-43a6-a30a-136f5a93140b","Type":"ContainerStarted","Data":"9af44b17412217a04314617772d3f6e416ba6bf8edca9e933651fc61a6aeced0"}
Sep 09 16:14:10.033559 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:10.033385    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"
Sep 09 16:14:10.035621 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:10.035589    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn" event={"ID":"2a904a14-212b-42ae-9b1f-ab0ef31a4bac","Type":"ContainerStarted","Data":"4dbff5e7f950993700bbdcffea4ec388b6d76dfccec77f1b3ee476c6194c0432"}
Sep 09 16:14:10.035763 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:10.035637    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn" event={"ID":"2a904a14-212b-42ae-9b1f-ab0ef31a4bac","Type":"ContainerStarted","Data":"f29872b8942d8eb866f1d55db59748d9dc0362a98b4cd2b76a4f6e1446870a16"}
Sep 09 16:14:10.035763 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:10.035660    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"
Sep 09 16:14:10.050051 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:10.050003    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6" podStartSLOduration=2.049981572 podStartE2EDuration="2.049981572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:10.047612722 +0000 UTC m=+1354.914123943" watchObservedRunningTime="2026-09-09 16:14:10.049981572 +0000 UTC m=+1354.916492797"
Sep 09 16:14:10.062478 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:10.062424    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn" podStartSLOduration=2.06240775 podStartE2EDuration="2.06240775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:10.061346458 +0000 UTC m=+1354.927857679" watchObservedRunningTime="2026-09-09 16:14:10.06240775 +0000 UTC m=+1354.928918972"
Sep 09 16:14:11.148732 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:11.148691    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:11.833707 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:11.833668    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:11.841449 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:11.841421    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"]
Sep 09 16:14:11.841657 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:11.841631    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg" podUID="ae7a5564-8f34-45e0-9ade-d2475ba94da3" containerName="payload-processing" containerID="cri-o://e4ace5c03d1b03f08d652223a0db576fb61a28346b969fd7f8c49492b5534415" gracePeriod=30
Sep 09 16:14:11.847227 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:11.847197    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"]
Sep 09 16:14:11.847415 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:11.847390    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg" podUID="7360aac7-5a06-4a34-bfa7-59fc0a98471e" containerName="payload-pre-processing" containerID="cri-o://3631926ebb412055bb728c64c1a6f7c8d6ca68f891374726a8d5cf8e33f22eab" gracePeriod=30
Sep 09 16:14:11.933367 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:11.933336    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:11.994085 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:11.994025    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:12.047875 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.047845    2556 generic.go:356] "Generic (PLEG): container finished" podID="ae7a5564-8f34-45e0-9ade-d2475ba94da3" containerID="e4ace5c03d1b03f08d652223a0db576fb61a28346b969fd7f8c49492b5534415" exitCode=0
Sep 09 16:14:12.048032 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.047912    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg" event={"ID":"ae7a5564-8f34-45e0-9ade-d2475ba94da3","Type":"ContainerDied","Data":"e4ace5c03d1b03f08d652223a0db576fb61a28346b969fd7f8c49492b5534415"}
Sep 09 16:14:12.050194 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.050160    2556 generic.go:356] "Generic (PLEG): container finished" podID="7360aac7-5a06-4a34-bfa7-59fc0a98471e" containerID="3631926ebb412055bb728c64c1a6f7c8d6ca68f891374726a8d5cf8e33f22eab" exitCode=0
Sep 09 16:14:12.050415 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.050387    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg" event={"ID":"7360aac7-5a06-4a34-bfa7-59fc0a98471e","Type":"ContainerDied","Data":"3631926ebb412055bb728c64c1a6f7c8d6ca68f891374726a8d5cf8e33f22eab"}
Sep 09 16:14:12.050571 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.050527    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:12.111580 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.111553    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"
Sep 09 16:14:12.114392 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.114373    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"
Sep 09 16:14:12.163790 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.163759    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7360aac7-5a06-4a34-bfa7-59fc0a98471e-plugins-config-volume\") pod \"7360aac7-5a06-4a34-bfa7-59fc0a98471e\" (UID: \"7360aac7-5a06-4a34-bfa7-59fc0a98471e\") "
Sep 09 16:14:12.163790 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.163794    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbc9t\" (UniqueName: \"kubernetes.io/projected/7360aac7-5a06-4a34-bfa7-59fc0a98471e-kube-api-access-mbc9t\") pod \"7360aac7-5a06-4a34-bfa7-59fc0a98471e\" (UID: \"7360aac7-5a06-4a34-bfa7-59fc0a98471e\") "
Sep 09 16:14:12.164291 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.163823    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7a5564-8f34-45e0-9ade-d2475ba94da3-plugins-config-volume\") pod \"ae7a5564-8f34-45e0-9ade-d2475ba94da3\" (UID: \"ae7a5564-8f34-45e0-9ade-d2475ba94da3\") "
Sep 09 16:14:12.164291 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.163852    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c28mw\" (UniqueName: \"kubernetes.io/projected/ae7a5564-8f34-45e0-9ade-d2475ba94da3-kube-api-access-c28mw\") pod \"ae7a5564-8f34-45e0-9ade-d2475ba94da3\" (UID: \"ae7a5564-8f34-45e0-9ade-d2475ba94da3\") "
Sep 09 16:14:12.164291 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.164137    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7360aac7-5a06-4a34-bfa7-59fc0a98471e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7360aac7-5a06-4a34-bfa7-59fc0a98471e" (UID: "7360aac7-5a06-4a34-bfa7-59fc0a98471e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:14:12.164291 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.164194    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7a5564-8f34-45e0-9ade-d2475ba94da3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae7a5564-8f34-45e0-9ade-d2475ba94da3" (UID: "ae7a5564-8f34-45e0-9ade-d2475ba94da3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:14:12.165990 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.165967    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7360aac7-5a06-4a34-bfa7-59fc0a98471e-kube-api-access-mbc9t" (OuterVolumeSpecName: "kube-api-access-mbc9t") pod "7360aac7-5a06-4a34-bfa7-59fc0a98471e" (UID: "7360aac7-5a06-4a34-bfa7-59fc0a98471e"). InnerVolumeSpecName "kube-api-access-mbc9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:14:12.166095 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.166076    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7a5564-8f34-45e0-9ade-d2475ba94da3-kube-api-access-c28mw" (OuterVolumeSpecName: "kube-api-access-c28mw") pod "ae7a5564-8f34-45e0-9ade-d2475ba94da3" (UID: "ae7a5564-8f34-45e0-9ade-d2475ba94da3"). InnerVolumeSpecName "kube-api-access-c28mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:14:12.264401 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.264365    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7360aac7-5a06-4a34-bfa7-59fc0a98471e-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:14:12.264401 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.264395    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbc9t\" (UniqueName: \"kubernetes.io/projected/7360aac7-5a06-4a34-bfa7-59fc0a98471e-kube-api-access-mbc9t\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:14:12.264401 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.264407    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7a5564-8f34-45e0-9ade-d2475ba94da3-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:14:12.264641 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.264420    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c28mw\" (UniqueName: \"kubernetes.io/projected/ae7a5564-8f34-45e0-9ade-d2475ba94da3-kube-api-access-c28mw\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:14:12.351954 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:12.351845    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:13.056112 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:13.056074    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg" event={"ID":"ae7a5564-8f34-45e0-9ade-d2475ba94da3","Type":"ContainerDied","Data":"98597fe14fb8c39b1ac0bf64e2fbd5dbe766f0f7bf01ddbb0daf7905481dc15f"}
Sep 09 16:14:13.056112 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:13.056091    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"
Sep 09 16:14:13.056358 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:13.056145    2556 scope.go:117] "RemoveContainer" containerID="e4ace5c03d1b03f08d652223a0db576fb61a28346b969fd7f8c49492b5534415"
Sep 09 16:14:13.057586 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:13.057504    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg" event={"ID":"7360aac7-5a06-4a34-bfa7-59fc0a98471e","Type":"ContainerDied","Data":"7caa706d6d6597d19e5bce6d1470a335399ad4c6fca46220554a448795506627"}
Sep 09 16:14:13.057586 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:13.057541    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"
Sep 09 16:14:13.066880 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:13.066861    2556 scope.go:117] "RemoveContainer" containerID="3631926ebb412055bb728c64c1a6f7c8d6ca68f891374726a8d5cf8e33f22eab"
Sep 09 16:14:13.084445 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:13.084419    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"]
Sep 09 16:14:13.092916 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:13.092892    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-677907eb-85cdf7ff9c-267vg"]
Sep 09 16:14:13.104534 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:13.104510    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"]
Sep 09 16:14:13.110676 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:13.110650    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-677907eb-6b75cc7c6b-hvgjg"]
Sep 09 16:14:13.763517 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:13.763454    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7360aac7-5a06-4a34-bfa7-59fc0a98471e" path="/var/lib/kubelet/pods/7360aac7-5a06-4a34-bfa7-59fc0a98471e/volumes"
Sep 09 16:14:13.764027 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:13.763998    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae7a5564-8f34-45e0-9ade-d2475ba94da3" path="/var/lib/kubelet/pods/ae7a5564-8f34-45e0-9ade-d2475ba94da3/volumes"
Sep 09 16:14:17.966248 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:17.966215    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"]
Sep 09 16:14:17.989913 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:17.985245    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:18.085353 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:18.085315    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:18.202717 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:18.202684    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:18.284037 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:18.283951    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:18.358318 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:18.358286    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:18.451806 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:18.451769    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:21.042319 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:21.042289    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"
Sep 09 16:14:21.042795 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:21.042606    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"
Sep 09 16:14:22.076862 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:22.076824    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:22.177112 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:22.177074    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:23.515103 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:23.515063    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:24.909569 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:24.909524    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:24.965678 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:24.965648    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:28.619954 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:28.619918    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:28.674458 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:28.674424    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:29.995338 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:29.995303    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:31.375344 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:31.375301    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:31.475120 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:31.475087    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:31.765339 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:31.765286    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:32.090395 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:32.090291    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:32.280795 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:32.280764    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:32.397872 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:32.397772    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:32.499162 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:32.499124    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:32.641634 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:32.641600    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:32.827717 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:32.827680    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:32.985823 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:32.985761    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp" podUID="4d9b272d-1123-455c-944b-bcc6f12710d9" containerName="main" containerID="cri-o://028fa1a71b2f9a69c62f5af85e7a3f3dc977c922f52a7a5280b835e2c17d93bc" gracePeriod=45
Sep 09 16:14:33.147428 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.147396    2556 generic.go:356] "Generic (PLEG): container finished" podID="4d9b272d-1123-455c-944b-bcc6f12710d9" containerID="028fa1a71b2f9a69c62f5af85e7a3f3dc977c922f52a7a5280b835e2c17d93bc" exitCode=0
Sep 09 16:14:33.147640 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.147482    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp" event={"ID":"4d9b272d-1123-455c-944b-bcc6f12710d9","Type":"ContainerDied","Data":"028fa1a71b2f9a69c62f5af85e7a3f3dc977c922f52a7a5280b835e2c17d93bc"}
Sep 09 16:14:33.231129 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.231107    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:14:33.347985 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.347949    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-model-cache\") pod \"4d9b272d-1123-455c-944b-bcc6f12710d9\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") "
Sep 09 16:14:33.348143 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.348024    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-dshm\") pod \"4d9b272d-1123-455c-944b-bcc6f12710d9\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") "
Sep 09 16:14:33.348143 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.348072    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-home\") pod \"4d9b272d-1123-455c-944b-bcc6f12710d9\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") "
Sep 09 16:14:33.348143 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.348107    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9b272d-1123-455c-944b-bcc6f12710d9-tls-certs\") pod \"4d9b272d-1123-455c-944b-bcc6f12710d9\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") "
Sep 09 16:14:33.348143 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.348132    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-tmp-dir\") pod \"4d9b272d-1123-455c-944b-bcc6f12710d9\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") "
Sep 09 16:14:33.348346 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.348159    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8j4b\" (UniqueName: \"kubernetes.io/projected/4d9b272d-1123-455c-944b-bcc6f12710d9-kube-api-access-n8j4b\") pod \"4d9b272d-1123-455c-944b-bcc6f12710d9\" (UID: \"4d9b272d-1123-455c-944b-bcc6f12710d9\") "
Sep 09 16:14:33.348346 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.348237    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d9b272d-1123-455c-944b-bcc6f12710d9" (UID: "4d9b272d-1123-455c-944b-bcc6f12710d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:14:33.348451 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.348389    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-home" (OuterVolumeSpecName: "home") pod "4d9b272d-1123-455c-944b-bcc6f12710d9" (UID: "4d9b272d-1123-455c-944b-bcc6f12710d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:14:33.348451 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.348412    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-model-cache\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:14:33.348451 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.348431    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d9b272d-1123-455c-944b-bcc6f12710d9" (UID: "4d9b272d-1123-455c-944b-bcc6f12710d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:14:33.350345 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.350314    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d9b272d-1123-455c-944b-bcc6f12710d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d9b272d-1123-455c-944b-bcc6f12710d9" (UID: "4d9b272d-1123-455c-944b-bcc6f12710d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:14:33.350345 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.350329    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-dshm" (OuterVolumeSpecName: "dshm") pod "4d9b272d-1123-455c-944b-bcc6f12710d9" (UID: "4d9b272d-1123-455c-944b-bcc6f12710d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:14:33.350546 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.350395    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9b272d-1123-455c-944b-bcc6f12710d9-kube-api-access-n8j4b" (OuterVolumeSpecName: "kube-api-access-n8j4b") pod "4d9b272d-1123-455c-944b-bcc6f12710d9" (UID: "4d9b272d-1123-455c-944b-bcc6f12710d9"). InnerVolumeSpecName "kube-api-access-n8j4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:14:33.449604 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.449571    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-dshm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:14:33.449604 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.449597    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-home\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:14:33.449604 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.449606    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9b272d-1123-455c-944b-bcc6f12710d9-tls-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:14:33.450041 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.449617    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9b272d-1123-455c-944b-bcc6f12710d9-tmp-dir\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:14:33.450041 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:33.449624    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8j4b\" (UniqueName: \"kubernetes.io/projected/4d9b272d-1123-455c-944b-bcc6f12710d9-kube-api-access-n8j4b\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:14:34.154163 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:34.154077    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"
Sep 09 16:14:34.154163 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:34.154094    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp" event={"ID":"4d9b272d-1123-455c-944b-bcc6f12710d9","Type":"ContainerDied","Data":"b8b722343cc7a00b0d68fc671ce7af62110dabbc48d735267643ad57fca19adb"}
Sep 09 16:14:34.154163 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:34.154138    2556 scope.go:117] "RemoveContainer" containerID="028fa1a71b2f9a69c62f5af85e7a3f3dc977c922f52a7a5280b835e2c17d93bc"
Sep 09 16:14:34.169610 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:34.169575    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"]
Sep 09 16:14:34.173475 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:34.173439    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-a372ad/noten-model-a372ad-kserve-7d6d6c4cdc-kbljp"]
Sep 09 16:14:35.762874 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:35.762851    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d9b272d-1123-455c-944b-bcc6f12710d9" path="/var/lib/kubelet/pods/4d9b272d-1123-455c-944b-bcc6f12710d9/volumes"
Sep 09 16:14:35.966861 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:35.966834    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:36.059298 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:36.059216    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:36.158952 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:36.158912    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:37.258894 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.258852    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:37.361594 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.361556    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:37.491015 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.490980    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:37.850133 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.850098    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:37.949137 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.949103    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:37.955592 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.955568    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"]
Sep 09 16:14:37.955861 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.955850    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae7a5564-8f34-45e0-9ade-d2475ba94da3" containerName="payload-processing"
Sep 09 16:14:37.955904 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.955863    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae7a5564-8f34-45e0-9ade-d2475ba94da3" containerName="payload-processing"
Sep 09 16:14:37.955904 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.955879    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7360aac7-5a06-4a34-bfa7-59fc0a98471e" containerName="payload-pre-processing"
Sep 09 16:14:37.955904 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.955885    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7360aac7-5a06-4a34-bfa7-59fc0a98471e" containerName="payload-pre-processing"
Sep 09 16:14:37.955904 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.955893    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9b272d-1123-455c-944b-bcc6f12710d9" containerName="main"
Sep 09 16:14:37.955904 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.955899    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9b272d-1123-455c-944b-bcc6f12710d9" containerName="main"
Sep 09 16:14:37.956057 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.955952    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae7a5564-8f34-45e0-9ade-d2475ba94da3" containerName="payload-processing"
Sep 09 16:14:37.956057 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.955960    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9b272d-1123-455c-944b-bcc6f12710d9" containerName="main"
Sep 09 16:14:37.956057 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.955966    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7360aac7-5a06-4a34-bfa7-59fc0a98471e" containerName="payload-pre-processing"
Sep 09 16:14:37.958946 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.958927    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"
Sep 09 16:14:37.961379 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.961354    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-13d59f-dockercfg-rtfhw\""
Sep 09 16:14:37.961524 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.961355    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-13d59f\""
Sep 09 16:14:37.973829 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.973806    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"]
Sep 09 16:14:37.993645 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.993619    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"]
Sep 09 16:14:37.996488 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:37.996456    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"
Sep 09 16:14:38.006868 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.006840    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"]
Sep 09 16:14:38.089875 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.089839    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b089384-dc8c-440c-aa46-dcce162ef93c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw\" (UID: \"1b089384-dc8c-440c-aa46-dcce162ef93c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"
Sep 09 16:14:38.089875 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.089877    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bprmg\" (UniqueName: \"kubernetes.io/projected/1b089384-dc8c-440c-aa46-dcce162ef93c-kube-api-access-bprmg\") pod \"payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw\" (UID: \"1b089384-dc8c-440c-aa46-dcce162ef93c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"
Sep 09 16:14:38.191373 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.191335    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b089384-dc8c-440c-aa46-dcce162ef93c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw\" (UID: \"1b089384-dc8c-440c-aa46-dcce162ef93c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"
Sep 09 16:14:38.191373 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.191373    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bprmg\" (UniqueName: \"kubernetes.io/projected/1b089384-dc8c-440c-aa46-dcce162ef93c-kube-api-access-bprmg\") pod \"payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw\" (UID: \"1b089384-dc8c-440c-aa46-dcce162ef93c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"
Sep 09 16:14:38.191629 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.191398    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2j2b\" (UniqueName: \"kubernetes.io/projected/35fc4078-ba08-4a23-a3f8-52463674c33d-kube-api-access-x2j2b\") pod \"payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r\" (UID: \"35fc4078-ba08-4a23-a3f8-52463674c33d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"
Sep 09 16:14:38.191629 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.191450    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35fc4078-ba08-4a23-a3f8-52463674c33d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r\" (UID: \"35fc4078-ba08-4a23-a3f8-52463674c33d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"
Sep 09 16:14:38.192048 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.192029    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b089384-dc8c-440c-aa46-dcce162ef93c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw\" (UID: \"1b089384-dc8c-440c-aa46-dcce162ef93c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"
Sep 09 16:14:38.200538 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.200520    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bprmg\" (UniqueName: \"kubernetes.io/projected/1b089384-dc8c-440c-aa46-dcce162ef93c-kube-api-access-bprmg\") pod \"payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw\" (UID: \"1b089384-dc8c-440c-aa46-dcce162ef93c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"
Sep 09 16:14:38.260838 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.260807    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:38.271693 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.271667    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"
Sep 09 16:14:38.291973 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.291948    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35fc4078-ba08-4a23-a3f8-52463674c33d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r\" (UID: \"35fc4078-ba08-4a23-a3f8-52463674c33d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"
Sep 09 16:14:38.292095 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.292016    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2j2b\" (UniqueName: \"kubernetes.io/projected/35fc4078-ba08-4a23-a3f8-52463674c33d-kube-api-access-x2j2b\") pod \"payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r\" (UID: \"35fc4078-ba08-4a23-a3f8-52463674c33d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"
Sep 09 16:14:38.292684 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.292660    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35fc4078-ba08-4a23-a3f8-52463674c33d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r\" (UID: \"35fc4078-ba08-4a23-a3f8-52463674c33d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"
Sep 09 16:14:38.300677 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.300655    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2j2b\" (UniqueName: \"kubernetes.io/projected/35fc4078-ba08-4a23-a3f8-52463674c33d-kube-api-access-x2j2b\") pod \"payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r\" (UID: \"35fc4078-ba08-4a23-a3f8-52463674c33d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"
Sep 09 16:14:38.306436 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.306414    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"
Sep 09 16:14:38.317820 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.317437    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:38.543734 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.543705    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"]
Sep 09 16:14:38.545994 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:14:38.545965    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b089384_dc8c_440c_aa46_dcce162ef93c.slice/crio-87d33c16fa56adfc88d6008fb50dfdc932f8623120e2d85a3f0b4643c68ec997 WatchSource:0}: Error finding container 87d33c16fa56adfc88d6008fb50dfdc932f8623120e2d85a3f0b4643c68ec997: Status 404 returned error can't find the container with id 87d33c16fa56adfc88d6008fb50dfdc932f8623120e2d85a3f0b4643c68ec997
Sep 09 16:14:38.582904 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.582878    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:38.595909 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.595850    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"]
Sep 09 16:14:38.597969 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:14:38.597948    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fc4078_ba08_4a23_a3f8_52463674c33d.slice/crio-c25591d1901ecf8d87d1137791304ac6ea8729c56610a538ef9429ece14375a2 WatchSource:0}: Error finding container c25591d1901ecf8d87d1137791304ac6ea8729c56610a538ef9429ece14375a2: Status 404 returned error can't find the container with id c25591d1901ecf8d87d1137791304ac6ea8729c56610a538ef9429ece14375a2
Sep 09 16:14:38.753294 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.753258    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:38.860862 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.860769    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:38.861354 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.861334    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"]
Sep 09 16:14:38.864122 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.864104    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"
Sep 09 16:14:38.866444 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.866408    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-b0ce96\""
Sep 09 16:14:38.866604 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.866476    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-b0ce96-dockercfg-d479x\""
Sep 09 16:14:38.875562 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.875533    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"]
Sep 09 16:14:38.887195 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.887172    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"]
Sep 09 16:14:38.890843 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.890823    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"
Sep 09 16:14:38.898770 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.898735    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"]
Sep 09 16:14:38.998382 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.998343    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec74dd7-e7eb-4405-83b4-fbd754814a39-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7\" (UID: \"7ec74dd7-e7eb-4405-83b4-fbd754814a39\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"
Sep 09 16:14:38.998561 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.998396    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzsct\" (UniqueName: \"kubernetes.io/projected/7ec74dd7-e7eb-4405-83b4-fbd754814a39-kube-api-access-tzsct\") pod \"payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7\" (UID: \"7ec74dd7-e7eb-4405-83b4-fbd754814a39\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"
Sep 09 16:14:38.998561 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.998455    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b26c81-5f3a-489d-95f9-55478cb22fa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw\" (UID: \"45b26c81-5f3a-489d-95f9-55478cb22fa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"
Sep 09 16:14:38.998561 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:38.998546    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv9fv\" (UniqueName: \"kubernetes.io/projected/45b26c81-5f3a-489d-95f9-55478cb22fa3-kube-api-access-qv9fv\") pod \"payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw\" (UID: \"45b26c81-5f3a-489d-95f9-55478cb22fa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"
Sep 09 16:14:39.099940 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.099901    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv9fv\" (UniqueName: \"kubernetes.io/projected/45b26c81-5f3a-489d-95f9-55478cb22fa3-kube-api-access-qv9fv\") pod \"payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw\" (UID: \"45b26c81-5f3a-489d-95f9-55478cb22fa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"
Sep 09 16:14:39.100105 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.099990    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec74dd7-e7eb-4405-83b4-fbd754814a39-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7\" (UID: \"7ec74dd7-e7eb-4405-83b4-fbd754814a39\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"
Sep 09 16:14:39.100105 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.100028    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzsct\" (UniqueName: \"kubernetes.io/projected/7ec74dd7-e7eb-4405-83b4-fbd754814a39-kube-api-access-tzsct\") pod \"payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7\" (UID: \"7ec74dd7-e7eb-4405-83b4-fbd754814a39\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"
Sep 09 16:14:39.100226 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.100150    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b26c81-5f3a-489d-95f9-55478cb22fa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw\" (UID: \"45b26c81-5f3a-489d-95f9-55478cb22fa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"
Sep 09 16:14:39.100730 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.100711    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec74dd7-e7eb-4405-83b4-fbd754814a39-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7\" (UID: \"7ec74dd7-e7eb-4405-83b4-fbd754814a39\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"
Sep 09 16:14:39.100997 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.100977    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b26c81-5f3a-489d-95f9-55478cb22fa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw\" (UID: \"45b26c81-5f3a-489d-95f9-55478cb22fa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"
Sep 09 16:14:39.107717 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.107689    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv9fv\" (UniqueName: \"kubernetes.io/projected/45b26c81-5f3a-489d-95f9-55478cb22fa3-kube-api-access-qv9fv\") pod \"payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw\" (UID: \"45b26c81-5f3a-489d-95f9-55478cb22fa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"
Sep 09 16:14:39.108049 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.108028    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzsct\" (UniqueName: \"kubernetes.io/projected/7ec74dd7-e7eb-4405-83b4-fbd754814a39-kube-api-access-tzsct\") pod \"payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7\" (UID: \"7ec74dd7-e7eb-4405-83b4-fbd754814a39\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"
Sep 09 16:14:39.174518 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.174492    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"
Sep 09 16:14:39.177851 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.177827    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw" event={"ID":"1b089384-dc8c-440c-aa46-dcce162ef93c","Type":"ContainerStarted","Data":"63bd422f5f68a6330f269302081ab60791d151c948efa0a594f335a1b727a611"}
Sep 09 16:14:39.177970 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.177883    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw" event={"ID":"1b089384-dc8c-440c-aa46-dcce162ef93c","Type":"ContainerStarted","Data":"87d33c16fa56adfc88d6008fb50dfdc932f8623120e2d85a3f0b4643c68ec997"}
Sep 09 16:14:39.177970 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.177916    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"
Sep 09 16:14:39.179442 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.179420    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r" event={"ID":"35fc4078-ba08-4a23-a3f8-52463674c33d","Type":"ContainerStarted","Data":"77652ad59dcf2cbcf2a7efd337353a17c260ce47b13170ff9631111b977e593a"}
Sep 09 16:14:39.179574 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.179479    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r" event={"ID":"35fc4078-ba08-4a23-a3f8-52463674c33d","Type":"ContainerStarted","Data":"c25591d1901ecf8d87d1137791304ac6ea8729c56610a538ef9429ece14375a2"}
Sep 09 16:14:39.179574 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.179510    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"
Sep 09 16:14:39.197776 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.197715    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw" podStartSLOduration=2.197696427 podStartE2EDuration="2.197696427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:39.196457639 +0000 UTC m=+1384.062968882" watchObservedRunningTime="2026-09-09 16:14:39.197696427 +0000 UTC m=+1384.064207649"
Sep 09 16:14:39.200953 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.200932    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"
Sep 09 16:14:39.264314 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.264272    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:39.452344 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.452235    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r" podStartSLOduration=2.452213564 podStartE2EDuration="2.452213564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:39.213989154 +0000 UTC m=+1384.080500375" watchObservedRunningTime="2026-09-09 16:14:39.452213564 +0000 UTC m=+1384.318724828"
Sep 09 16:14:39.453714 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.453648    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"]
Sep 09 16:14:39.456106 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:14:39.456080    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b26c81_5f3a_489d_95f9_55478cb22fa3.slice/crio-ddda6e8850c1bc3dd805f07024764b7dad63902d0ea3dd19d43fc32f9a39f84a WatchSource:0}: Error finding container ddda6e8850c1bc3dd805f07024764b7dad63902d0ea3dd19d43fc32f9a39f84a: Status 404 returned error can't find the container with id ddda6e8850c1bc3dd805f07024764b7dad63902d0ea3dd19d43fc32f9a39f84a
Sep 09 16:14:39.479901 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:39.479865    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"]
Sep 09 16:14:39.488510 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:14:39.488481    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec74dd7_e7eb_4405_83b4_fbd754814a39.slice/crio-cf011deea190b45718adb6298ab5a52f3829d4a6eaee8305049b1aa85e5533af WatchSource:0}: Error finding container cf011deea190b45718adb6298ab5a52f3829d4a6eaee8305049b1aa85e5533af: Status 404 returned error can't find the container with id cf011deea190b45718adb6298ab5a52f3829d4a6eaee8305049b1aa85e5533af
Sep 09 16:14:40.028277 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:40.028239    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:40.186042 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:40.186003    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7" event={"ID":"7ec74dd7-e7eb-4405-83b4-fbd754814a39","Type":"ContainerStarted","Data":"773d507526b9d7db58990e0480b5bf4729a91d2c46ee2128b0d8227f8e1525c8"}
Sep 09 16:14:40.186229 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:40.186055    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7" event={"ID":"7ec74dd7-e7eb-4405-83b4-fbd754814a39","Type":"ContainerStarted","Data":"cf011deea190b45718adb6298ab5a52f3829d4a6eaee8305049b1aa85e5533af"}
Sep 09 16:14:40.186229 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:40.186213    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"
Sep 09 16:14:40.187517 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:40.187493    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw" event={"ID":"45b26c81-5f3a-489d-95f9-55478cb22fa3","Type":"ContainerStarted","Data":"014bdabfc7c16c0afda84ea9eca3df7f3813f188ece0ecf227bef3be77f903a5"}
Sep 09 16:14:40.187646 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:40.187539    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"
Sep 09 16:14:40.187646 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:40.187562    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw" event={"ID":"45b26c81-5f3a-489d-95f9-55478cb22fa3","Type":"ContainerStarted","Data":"ddda6e8850c1bc3dd805f07024764b7dad63902d0ea3dd19d43fc32f9a39f84a"}
Sep 09 16:14:40.206162 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:40.206111    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7" podStartSLOduration=2.206092387 podStartE2EDuration="2.206092387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:40.203062877 +0000 UTC m=+1385.069574108" watchObservedRunningTime="2026-09-09 16:14:40.206092387 +0000 UTC m=+1385.072603603"
Sep 09 16:14:40.219608 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:40.219565    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw" podStartSLOduration=2.21955187 podStartE2EDuration="2.21955187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:40.218320175 +0000 UTC m=+1385.084831398" watchObservedRunningTime="2026-09-09 16:14:40.21955187 +0000 UTC m=+1385.086063102"
Sep 09 16:14:50.190090 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:50.190051    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"
Sep 09 16:14:50.190499 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:50.190138    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"
Sep 09 16:14:51.195138 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:51.195100    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"
Sep 09 16:14:51.195617 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:51.195175    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"
Sep 09 16:14:58.245995 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:58.245954    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:14:59.655510 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:14:59.655457    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:00.143902 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:00.143862    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0ce96-29816175-nknzh"]
Sep 09 16:15:00.148476 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:00.148442    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816175-trxmj"]
Sep 09 16:15:00.148578 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:00.148560    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0ce96-29816175-nknzh"
Sep 09 16:15:00.150795 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:00.150775    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\""
Sep 09 16:15:00.151724 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:00.151699    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816175-trxmj"
Sep 09 16:15:00.153767 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:00.153746    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0ce96-29816175-nknzh"]
Sep 09 16:15:00.162352 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:00.162334    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0ce96-29816175-nknzh"
Sep 09 16:15:00.170419 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:00.170394    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816175-trxmj"]
Sep 09 16:15:00.170837 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:00.170818    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816175-trxmj"
Sep 09 16:15:00.444347 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:00.444266    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0ce96-29816175-nknzh"]
Sep 09 16:15:00.446807 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:15:00.446779    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0169b228_de36_40c9_830b_89fe5a5eaf0c.slice/crio-f973839c6f1078e7554f61dcc431e63ee74967459280ec565c3b6a07e8013a23 WatchSource:0}: Error finding container f973839c6f1078e7554f61dcc431e63ee74967459280ec565c3b6a07e8013a23: Status 404 returned error can't find the container with id f973839c6f1078e7554f61dcc431e63ee74967459280ec565c3b6a07e8013a23
Sep 09 16:15:00.455128 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:00.455099    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816175-trxmj"]
Sep 09 16:15:00.459336 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:15:00.459311    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa9cdce_7e7f_4c88_a75c_7aaf5d98c5f7.slice/crio-74f6af97222698c08a444d6c8c676c4703d04614e3a05672425cbc7c97178ce1 WatchSource:0}: Error finding container 74f6af97222698c08a444d6c8c676c4703d04614e3a05672425cbc7c97178ce1: Status 404 returned error can't find the container with id 74f6af97222698c08a444d6c8c676c4703d04614e3a05672425cbc7c97178ce1
Sep 09 16:15:00.481870 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:00.481844    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:00.591912 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:00.591873    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:00.964636 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:00.964548    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:01.065547 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:01.065494    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:01.294292 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:01.294193    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816175-trxmj" event={"ID":"6aa9cdce-7e7f-4c88-a75c-7aaf5d98c5f7","Type":"ContainerStarted","Data":"fbb5c3f3410c3cc03aaf691712fc52cc0dca60ec9c944f4835d58130bb5c21c1"}
Sep 09 16:15:01.294499 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:01.294387    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816175-trxmj" event={"ID":"6aa9cdce-7e7f-4c88-a75c-7aaf5d98c5f7","Type":"ContainerStarted","Data":"74f6af97222698c08a444d6c8c676c4703d04614e3a05672425cbc7c97178ce1"}
Sep 09 16:15:01.295839 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:01.295814    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0ce96-29816175-nknzh" event={"ID":"0169b228-de36-40c9-830b-89fe5a5eaf0c","Type":"ContainerStarted","Data":"ba3a5ff04835a86f0980f0d99fec5b5a1845f5853ed156c5cebe236555b95356"}
Sep 09 16:15:01.295956 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:01.295859    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0ce96-29816175-nknzh" event={"ID":"0169b228-de36-40c9-830b-89fe5a5eaf0c","Type":"ContainerStarted","Data":"f973839c6f1078e7554f61dcc431e63ee74967459280ec565c3b6a07e8013a23"}
Sep 09 16:15:01.309263 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:01.309227    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:01.319762 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:01.319713    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816175-trxmj" podStartSLOduration=1.319696625 podStartE2EDuration="1.319696625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:01.317741251 +0000 UTC m=+1406.184252471" watchObservedRunningTime="2026-09-09 16:15:01.319696625 +0000 UTC m=+1406.186207845"
Sep 09 16:15:01.406178 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:01.406129    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:01.644808 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:01.644721    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:02.002627 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:02.002592    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:02.302159 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:02.302067    2556 generic.go:356] "Generic (PLEG): container finished" podID="6aa9cdce-7e7f-4c88-a75c-7aaf5d98c5f7" containerID="fbb5c3f3410c3cc03aaf691712fc52cc0dca60ec9c944f4835d58130bb5c21c1" exitCode=0
Sep 09 16:15:02.302326 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:02.302160    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816175-trxmj" event={"ID":"6aa9cdce-7e7f-4c88-a75c-7aaf5d98c5f7","Type":"ContainerDied","Data":"fbb5c3f3410c3cc03aaf691712fc52cc0dca60ec9c944f4835d58130bb5c21c1"}
Sep 09 16:15:02.303724 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:02.303698    2556 generic.go:356] "Generic (PLEG): container finished" podID="0169b228-de36-40c9-830b-89fe5a5eaf0c" containerID="ba3a5ff04835a86f0980f0d99fec5b5a1845f5853ed156c5cebe236555b95356" exitCode=0
Sep 09 16:15:02.303839 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:02.303748    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0ce96-29816175-nknzh" event={"ID":"0169b228-de36-40c9-830b-89fe5a5eaf0c","Type":"ContainerDied","Data":"ba3a5ff04835a86f0980f0d99fec5b5a1845f5853ed156c5cebe236555b95356"}
Sep 09 16:15:02.319400 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:02.319347    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0ce96-29816175-nknzh" podStartSLOduration=2.319331208 podStartE2EDuration="2.319331208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:01.341222719 +0000 UTC m=+1406.207733966" watchObservedRunningTime="2026-09-09 16:15:02.319331208 +0000 UTC m=+1407.185842472"
Sep 09 16:15:03.463380 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:03.463355    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0ce96-29816175-nknzh"
Sep 09 16:15:03.466159 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:03.466142    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816175-trxmj"
Sep 09 16:15:04.314383 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:04.314352    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816175-trxmj"
Sep 09 16:15:04.314571 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:04.314386    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816175-trxmj" event={"ID":"6aa9cdce-7e7f-4c88-a75c-7aaf5d98c5f7","Type":"ContainerDied","Data":"74f6af97222698c08a444d6c8c676c4703d04614e3a05672425cbc7c97178ce1"}
Sep 09 16:15:04.314571 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:04.314430    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74f6af97222698c08a444d6c8c676c4703d04614e3a05672425cbc7c97178ce1"
Sep 09 16:15:04.315631 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:04.315606    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0ce96-29816175-nknzh" event={"ID":"0169b228-de36-40c9-830b-89fe5a5eaf0c","Type":"ContainerDied","Data":"f973839c6f1078e7554f61dcc431e63ee74967459280ec565c3b6a07e8013a23"}
Sep 09 16:15:04.315749 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:04.315652    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f973839c6f1078e7554f61dcc431e63ee74967459280ec565c3b6a07e8013a23"
Sep 09 16:15:04.315749 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:04.315612    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0ce96-29816175-nknzh"
Sep 09 16:15:05.206707 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:05.206673    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:05.306670 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:05.306636    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:06.492256 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:06.492215    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:06.591241 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:06.591204    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:06.758211 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:06.758126    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:06.950225 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:06.950182    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"]
Sep 09 16:15:06.950656 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:06.950636    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa9cdce-7e7f-4c88-a75c-7aaf5d98c5f7" containerName="cleanup"
Sep 09 16:15:06.950656 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:06.950658    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa9cdce-7e7f-4c88-a75c-7aaf5d98c5f7" containerName="cleanup"
Sep 09 16:15:06.950824 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:06.950677    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0169b228-de36-40c9-830b-89fe5a5eaf0c" containerName="cleanup"
Sep 09 16:15:06.950824 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:06.950685    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0169b228-de36-40c9-830b-89fe5a5eaf0c" containerName="cleanup"
Sep 09 16:15:06.950824 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:06.950759    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa9cdce-7e7f-4c88-a75c-7aaf5d98c5f7" containerName="cleanup"
Sep 09 16:15:06.950824 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:06.950772    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0169b228-de36-40c9-830b-89fe5a5eaf0c" containerName="cleanup"
Sep 09 16:15:06.953645 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:06.953624    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"
Sep 09 16:15:06.955832 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:06.955809    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-008161-dockercfg-r88g4\""
Sep 09 16:15:06.955938 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:06.955837    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-008161\""
Sep 09 16:15:06.964111 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:06.964086    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"]
Sep 09 16:15:07.027543 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:07.027423    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11392f7-b713-4d01-8a3c-c9e865560acd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt\" (UID: \"d11392f7-b713-4d01-8a3c-c9e865560acd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"
Sep 09 16:15:07.027695 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:07.027617    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl66s\" (UniqueName: \"kubernetes.io/projected/d11392f7-b713-4d01-8a3c-c9e865560acd-kube-api-access-gl66s\") pod \"payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt\" (UID: \"d11392f7-b713-4d01-8a3c-c9e865560acd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"
Sep 09 16:15:07.128578 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:07.128534    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl66s\" (UniqueName: \"kubernetes.io/projected/d11392f7-b713-4d01-8a3c-c9e865560acd-kube-api-access-gl66s\") pod \"payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt\" (UID: \"d11392f7-b713-4d01-8a3c-c9e865560acd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"
Sep 09 16:15:07.128773 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:07.128590    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11392f7-b713-4d01-8a3c-c9e865560acd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt\" (UID: \"d11392f7-b713-4d01-8a3c-c9e865560acd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"
Sep 09 16:15:07.129260 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:07.129242    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11392f7-b713-4d01-8a3c-c9e865560acd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt\" (UID: \"d11392f7-b713-4d01-8a3c-c9e865560acd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"
Sep 09 16:15:07.136929 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:07.136894    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl66s\" (UniqueName: \"kubernetes.io/projected/d11392f7-b713-4d01-8a3c-c9e865560acd-kube-api-access-gl66s\") pod \"payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt\" (UID: \"d11392f7-b713-4d01-8a3c-c9e865560acd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"
Sep 09 16:15:07.169984 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:07.169939    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:07.264797 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:07.264742    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"
Sep 09 16:15:07.535183 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:07.535147    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"]
Sep 09 16:15:07.537433 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:15:07.537401    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11392f7_b713_4d01_8a3c_c9e865560acd.slice/crio-ec891e025e4b24df381e2c84d989d797165139be39500e45cf3a599c065fbc48 WatchSource:0}: Error finding container ec891e025e4b24df381e2c84d989d797165139be39500e45cf3a599c065fbc48: Status 404 returned error can't find the container with id ec891e025e4b24df381e2c84d989d797165139be39500e45cf3a599c065fbc48
Sep 09 16:15:08.339634 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:08.339598    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt" event={"ID":"d11392f7-b713-4d01-8a3c-c9e865560acd","Type":"ContainerStarted","Data":"440b333b06dbe901e0252d5bb5310677c78f2f1d6c28aea157501fc1d6ca0630"}
Sep 09 16:15:08.339814 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:08.339666    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"
Sep 09 16:15:08.339814 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:08.339690    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt" event={"ID":"d11392f7-b713-4d01-8a3c-c9e865560acd","Type":"ContainerStarted","Data":"ec891e025e4b24df381e2c84d989d797165139be39500e45cf3a599c065fbc48"}
Sep 09 16:15:08.355083 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:08.355035    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt" podStartSLOduration=2.355015614 podStartE2EDuration="2.355015614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:08.354613297 +0000 UTC m=+1413.221124520" watchObservedRunningTime="2026-09-09 16:15:08.355015614 +0000 UTC m=+1413.221526835"
Sep 09 16:15:19.164297 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:19.164260    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:19.346448 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:19.346421    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"
Sep 09 16:15:25.268707 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:25.268673    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:25.371241 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:25.371203    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:30.201496 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:30.201387    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:31.001598 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:31.001557    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7695-4ec61e"]
Sep 09 16:15:31.006435 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:31.006414    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-4ec61e"
Sep 09 16:15:31.009146 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:31.008918    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lp4n9\""
Sep 09 16:15:31.011373 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:31.011347    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-4ec61e"]
Sep 09 16:15:31.137550 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:31.137511    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th95v\" (UniqueName: \"kubernetes.io/projected/5c9d44dd-c61b-4597-89e8-507d06de8bf1-kube-api-access-th95v\") pod \"test-curl-7695-4ec61e\" (UID: \"5c9d44dd-c61b-4597-89e8-507d06de8bf1\") " pod="opendatahub/test-curl-7695-4ec61e"
Sep 09 16:15:31.238756 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:31.238720    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th95v\" (UniqueName: \"kubernetes.io/projected/5c9d44dd-c61b-4597-89e8-507d06de8bf1-kube-api-access-th95v\") pod \"test-curl-7695-4ec61e\" (UID: \"5c9d44dd-c61b-4597-89e8-507d06de8bf1\") " pod="opendatahub/test-curl-7695-4ec61e"
Sep 09 16:15:31.247034 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:31.247007    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th95v\" (UniqueName: \"kubernetes.io/projected/5c9d44dd-c61b-4597-89e8-507d06de8bf1-kube-api-access-th95v\") pod \"test-curl-7695-4ec61e\" (UID: \"5c9d44dd-c61b-4597-89e8-507d06de8bf1\") " pod="opendatahub/test-curl-7695-4ec61e"
Sep 09 16:15:31.322001 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:31.321908    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-4ec61e"
Sep 09 16:15:31.611282 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:31.611253    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-4ec61e"]
Sep 09 16:15:31.613883 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:15:31.613845    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c9d44dd_c61b_4597_89e8_507d06de8bf1.slice/crio-9599ff13d41b2b9d5dd59b6f852b4dd39cf8074633ac181de8a619bb8bcefff3 WatchSource:0}: Error finding container 9599ff13d41b2b9d5dd59b6f852b4dd39cf8074633ac181de8a619bb8bcefff3: Status 404 returned error can't find the container with id 9599ff13d41b2b9d5dd59b6f852b4dd39cf8074633ac181de8a619bb8bcefff3
Sep 09 16:15:32.453230 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:32.453194    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-4ec61e" event={"ID":"5c9d44dd-c61b-4597-89e8-507d06de8bf1","Type":"ContainerStarted","Data":"91edf56fb61fe067deaba2b95e78e6c5d429cdc51bab2181835e20fe75509024"}
Sep 09 16:15:32.453681 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:32.453238    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-4ec61e" event={"ID":"5c9d44dd-c61b-4597-89e8-507d06de8bf1","Type":"ContainerStarted","Data":"9599ff13d41b2b9d5dd59b6f852b4dd39cf8074633ac181de8a619bb8bcefff3"}
Sep 09 16:15:32.468449 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:32.468405    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7695-4ec61e" podStartSLOduration=2.468389706 podStartE2EDuration="2.468389706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:32.466695558 +0000 UTC m=+1437.333206779" watchObservedRunningTime="2026-09-09 16:15:32.468389706 +0000 UTC m=+1437.334900926"
Sep 09 16:15:33.080269 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:33.080234    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7695-4ec61e"]
Sep 09 16:15:33.084877 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:33.084851    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7695-4ec61e"]
Sep 09 16:15:33.207770 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:33.207735    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7695-5d3e44"]
Sep 09 16:15:33.208085 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:33.208073    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c9d44dd-c61b-4597-89e8-507d06de8bf1" containerName="test-curl-7695-4ec61e"
Sep 09 16:15:33.208145 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:33.208087    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c9d44dd-c61b-4597-89e8-507d06de8bf1" containerName="test-curl-7695-4ec61e"
Sep 09 16:15:33.208145 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:33.208138    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c9d44dd-c61b-4597-89e8-507d06de8bf1" containerName="test-curl-7695-4ec61e"
Sep 09 16:15:33.211227 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:33.211209    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-5d3e44"
Sep 09 16:15:33.216855 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:33.216832    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-5d3e44"]
Sep 09 16:15:33.357482 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:33.357429    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7dqx\" (UniqueName: \"kubernetes.io/projected/edb26280-ab77-4822-8ee1-1cb90b963e59-kube-api-access-h7dqx\") pod \"test-curl-7695-5d3e44\" (UID: \"edb26280-ab77-4822-8ee1-1cb90b963e59\") " pod="opendatahub/test-curl-7695-5d3e44"
Sep 09 16:15:33.457718 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:15:33.457688    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7695-4ec61e" container="test-curl-7695-4ec61e"
Sep 09 16:15:33.458150 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:33.458105    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7dqx\" (UniqueName: \"kubernetes.io/projected/edb26280-ab77-4822-8ee1-1cb90b963e59-kube-api-access-h7dqx\") pod \"test-curl-7695-5d3e44\" (UID: \"edb26280-ab77-4822-8ee1-1cb90b963e59\") " pod="opendatahub/test-curl-7695-5d3e44"
Sep 09 16:15:33.466154 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:33.466127    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7dqx\" (UniqueName: \"kubernetes.io/projected/edb26280-ab77-4822-8ee1-1cb90b963e59-kube-api-access-h7dqx\") pod \"test-curl-7695-5d3e44\" (UID: \"edb26280-ab77-4822-8ee1-1cb90b963e59\") " pod="opendatahub/test-curl-7695-5d3e44"
Sep 09 16:15:33.521215 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:33.521184    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-5d3e44"
Sep 09 16:15:33.787765 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:33.787741    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-5d3e44"]
Sep 09 16:15:33.789863 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:15:33.789832    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedb26280_ab77_4822_8ee1_1cb90b963e59.slice/crio-99974656e0dee30c00f23dc01174a09b3de735225d2926424c7c4f16a33df7c5 WatchSource:0}: Error finding container 99974656e0dee30c00f23dc01174a09b3de735225d2926424c7c4f16a33df7c5: Status 404 returned error can't find the container with id 99974656e0dee30c00f23dc01174a09b3de735225d2926424c7c4f16a33df7c5
Sep 09 16:15:34.463541 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:34.463502    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-5d3e44" event={"ID":"edb26280-ab77-4822-8ee1-1cb90b963e59","Type":"ContainerStarted","Data":"60048c4cb282fa30d1158b05b06da6f7d255c1d24adb576b1db1b2280106b9dc"}
Sep 09 16:15:34.464001 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:34.463556    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-5d3e44" event={"ID":"edb26280-ab77-4822-8ee1-1cb90b963e59","Type":"ContainerStarted","Data":"99974656e0dee30c00f23dc01174a09b3de735225d2926424c7c4f16a33df7c5"}
Sep 09 16:15:34.464001 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:34.463643    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7695-4ec61e" podUID="5c9d44dd-c61b-4597-89e8-507d06de8bf1" containerName="test-curl-7695-4ec61e" containerID="cri-o://91edf56fb61fe067deaba2b95e78e6c5d429cdc51bab2181835e20fe75509024" gracePeriod=2
Sep 09 16:15:34.476973 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:34.476927    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7695-5d3e44" podStartSLOduration=1.476913684 podStartE2EDuration="1.476913684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:34.475897649 +0000 UTC m=+1439.342408870" watchObservedRunningTime="2026-09-09 16:15:34.476913684 +0000 UTC m=+1439.343424904"
Sep 09 16:15:35.108566 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:35.108524    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7695-5d3e44"]
Sep 09 16:15:35.111482 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:35.111430    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7695-5d3e44"]
Sep 09 16:15:35.233578 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:35.233542    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7695-e6bc38"]
Sep 09 16:15:35.234006 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:35.233988    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb26280-ab77-4822-8ee1-1cb90b963e59" containerName="test-curl-7695-5d3e44"
Sep 09 16:15:35.234102 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:35.234010    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb26280-ab77-4822-8ee1-1cb90b963e59" containerName="test-curl-7695-5d3e44"
Sep 09 16:15:35.234102 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:35.234090    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb26280-ab77-4822-8ee1-1cb90b963e59" containerName="test-curl-7695-5d3e44"
Sep 09 16:15:35.237206 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:35.237185    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-e6bc38"
Sep 09 16:15:35.242660 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:35.242636    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-e6bc38"]
Sep 09 16:15:35.375401 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:35.375369    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rc9g\" (UniqueName: \"kubernetes.io/projected/eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c-kube-api-access-9rc9g\") pod \"test-curl-7695-e6bc38\" (UID: \"eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c\") " pod="opendatahub/test-curl-7695-e6bc38"
Sep 09 16:15:35.468593 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:15:35.468566    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7695-5d3e44" container="test-curl-7695-5d3e44"
Sep 09 16:15:35.476105 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:35.476080    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rc9g\" (UniqueName: \"kubernetes.io/projected/eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c-kube-api-access-9rc9g\") pod \"test-curl-7695-e6bc38\" (UID: \"eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c\") " pod="opendatahub/test-curl-7695-e6bc38"
Sep 09 16:15:35.483984 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:35.483934    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rc9g\" (UniqueName: \"kubernetes.io/projected/eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c-kube-api-access-9rc9g\") pod \"test-curl-7695-e6bc38\" (UID: \"eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c\") " pod="opendatahub/test-curl-7695-e6bc38"
Sep 09 16:15:35.547369 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:35.547325    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-e6bc38"
Sep 09 16:15:35.818191 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:35.818110    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-e6bc38"]
Sep 09 16:15:35.821796 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:15:35.821762    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb87cf21_f6a4_482b_8c50_e6c91b2ccc1c.slice/crio-91a3b6915760e900ccfe1165bea7b468f4260ad290c6c74a8e2b56e562452221 WatchSource:0}: Error finding container 91a3b6915760e900ccfe1165bea7b468f4260ad290c6c74a8e2b56e562452221: Status 404 returned error can't find the container with id 91a3b6915760e900ccfe1165bea7b468f4260ad290c6c74a8e2b56e562452221
Sep 09 16:15:36.474615 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:36.474581    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-e6bc38" event={"ID":"eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c","Type":"ContainerStarted","Data":"c088eab9c905cac357ffd562d76000da8acc2cf636483db38997640f72ffe5f9"}
Sep 09 16:15:36.474919 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:36.474632    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-e6bc38" event={"ID":"eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c","Type":"ContainerStarted","Data":"91a3b6915760e900ccfe1165bea7b468f4260ad290c6c74a8e2b56e562452221"}
Sep 09 16:15:36.474919 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:36.474755    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7695-5d3e44" podUID="edb26280-ab77-4822-8ee1-1cb90b963e59" containerName="test-curl-7695-5d3e44" containerID="cri-o://60048c4cb282fa30d1158b05b06da6f7d255c1d24adb576b1db1b2280106b9dc" gracePeriod=2
Sep 09 16:15:36.489796 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:36.489750    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7695-e6bc38" podStartSLOduration=1.489733671 podStartE2EDuration="1.489733671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:36.488323673 +0000 UTC m=+1441.354834915" watchObservedRunningTime="2026-09-09 16:15:36.489733671 +0000 UTC m=+1441.356244905"
Sep 09 16:15:36.649161 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:36.649137    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-4ec61e"
Sep 09 16:15:36.786777 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:36.786677    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th95v\" (UniqueName: \"kubernetes.io/projected/5c9d44dd-c61b-4597-89e8-507d06de8bf1-kube-api-access-th95v\") pod \"5c9d44dd-c61b-4597-89e8-507d06de8bf1\" (UID: \"5c9d44dd-c61b-4597-89e8-507d06de8bf1\") "
Sep 09 16:15:36.788787 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:36.788759    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9d44dd-c61b-4597-89e8-507d06de8bf1-kube-api-access-th95v" (OuterVolumeSpecName: "kube-api-access-th95v") pod "5c9d44dd-c61b-4597-89e8-507d06de8bf1" (UID: "5c9d44dd-c61b-4597-89e8-507d06de8bf1"). InnerVolumeSpecName "kube-api-access-th95v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:15:36.887775 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:36.887743    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th95v\" (UniqueName: \"kubernetes.io/projected/5c9d44dd-c61b-4597-89e8-507d06de8bf1-kube-api-access-th95v\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:15:37.088205 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.088081    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7695-e6bc38"]
Sep 09 16:15:37.090332 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.090302    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7695-e6bc38"]
Sep 09 16:15:37.205411 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.205360    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7695-aef458"]
Sep 09 16:15:37.205809 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.205789    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c" containerName="test-curl-7695-e6bc38"
Sep 09 16:15:37.205891 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.205812    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c" containerName="test-curl-7695-e6bc38"
Sep 09 16:15:37.205953 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.205925    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c" containerName="test-curl-7695-e6bc38"
Sep 09 16:15:37.208892 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.208870    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-aef458"
Sep 09 16:15:37.214562 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.214529    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-aef458"]
Sep 09 16:15:37.292098 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.292050    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dksbn\" (UniqueName: \"kubernetes.io/projected/6a8bd5ff-4cab-4fe4-a448-964375cce63b-kube-api-access-dksbn\") pod \"test-curl-7695-aef458\" (UID: \"6a8bd5ff-4cab-4fe4-a448-964375cce63b\") " pod="opendatahub/test-curl-7695-aef458"
Sep 09 16:15:37.392767 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.392677    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dksbn\" (UniqueName: \"kubernetes.io/projected/6a8bd5ff-4cab-4fe4-a448-964375cce63b-kube-api-access-dksbn\") pod \"test-curl-7695-aef458\" (UID: \"6a8bd5ff-4cab-4fe4-a448-964375cce63b\") " pod="opendatahub/test-curl-7695-aef458"
Sep 09 16:15:37.401560 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.401531    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dksbn\" (UniqueName: \"kubernetes.io/projected/6a8bd5ff-4cab-4fe4-a448-964375cce63b-kube-api-access-dksbn\") pod \"test-curl-7695-aef458\" (UID: \"6a8bd5ff-4cab-4fe4-a448-964375cce63b\") " pod="opendatahub/test-curl-7695-aef458"
Sep 09 16:15:37.480789 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.480755    2556 generic.go:356] "Generic (PLEG): container finished" podID="5c9d44dd-c61b-4597-89e8-507d06de8bf1" containerID="91edf56fb61fe067deaba2b95e78e6c5d429cdc51bab2181835e20fe75509024" exitCode=137
Sep 09 16:15:37.481226 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.480816    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-4ec61e"
Sep 09 16:15:37.481226 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.480858    2556 scope.go:117] "RemoveContainer" containerID="91edf56fb61fe067deaba2b95e78e6c5d429cdc51bab2181835e20fe75509024"
Sep 09 16:15:37.481226 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:15:37.481166    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7695-e6bc38" container="test-curl-7695-e6bc38"
Sep 09 16:15:37.490225 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.490207    2556 scope.go:117] "RemoveContainer" containerID="91edf56fb61fe067deaba2b95e78e6c5d429cdc51bab2181835e20fe75509024"
Sep 09 16:15:37.490538 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:15:37.490517    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91edf56fb61fe067deaba2b95e78e6c5d429cdc51bab2181835e20fe75509024\": container with ID starting with 91edf56fb61fe067deaba2b95e78e6c5d429cdc51bab2181835e20fe75509024 not found: ID does not exist" containerID="91edf56fb61fe067deaba2b95e78e6c5d429cdc51bab2181835e20fe75509024"
Sep 09 16:15:37.490615 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.490544    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91edf56fb61fe067deaba2b95e78e6c5d429cdc51bab2181835e20fe75509024"} err="failed to get container status \"91edf56fb61fe067deaba2b95e78e6c5d429cdc51bab2181835e20fe75509024\": rpc error: code = NotFound desc = could not find container \"91edf56fb61fe067deaba2b95e78e6c5d429cdc51bab2181835e20fe75509024\": container with ID starting with 91edf56fb61fe067deaba2b95e78e6c5d429cdc51bab2181835e20fe75509024 not found: ID does not exist"
Sep 09 16:15:37.492116 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:15:37.492094    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7695-4ec61e\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="5c9d44dd-c61b-4597-89e8-507d06de8bf1" pod="opendatahub/test-curl-7695-4ec61e"
Sep 09 16:15:37.518270 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.518249    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-aef458"
Sep 09 16:15:37.763420 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.763386    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c9d44dd-c61b-4597-89e8-507d06de8bf1" path="/var/lib/kubelet/pods/5c9d44dd-c61b-4597-89e8-507d06de8bf1/volumes"
Sep 09 16:15:37.786307 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:37.786175    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-aef458"]
Sep 09 16:15:37.788952 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:15:37.788923    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a8bd5ff_4cab_4fe4_a448_964375cce63b.slice/crio-4d9f2b673a1842c55cd4f168acffe02ee4984a3c6ff0760f5da47c2b9bf39c22 WatchSource:0}: Error finding container 4d9f2b673a1842c55cd4f168acffe02ee4984a3c6ff0760f5da47c2b9bf39c22: Status 404 returned error can't find the container with id 4d9f2b673a1842c55cd4f168acffe02ee4984a3c6ff0760f5da47c2b9bf39c22
Sep 09 16:15:38.487271 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:38.487240    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-aef458" event={"ID":"6a8bd5ff-4cab-4fe4-a448-964375cce63b","Type":"ContainerStarted","Data":"979598ad6f7b6dbaa77802b266d0659f70139d0430a17e58440fb60def2ccbca"}
Sep 09 16:15:38.487639 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:38.487286    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-aef458" event={"ID":"6a8bd5ff-4cab-4fe4-a448-964375cce63b","Type":"ContainerStarted","Data":"4d9f2b673a1842c55cd4f168acffe02ee4984a3c6ff0760f5da47c2b9bf39c22"}
Sep 09 16:15:38.488598 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:38.488572    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7695-e6bc38" podUID="eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c" containerName="test-curl-7695-e6bc38" containerID="cri-o://c088eab9c905cac357ffd562d76000da8acc2cf636483db38997640f72ffe5f9" gracePeriod=2
Sep 09 16:15:38.501912 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:38.501866    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7695-aef458" podStartSLOduration=1.501850184 podStartE2EDuration="1.501850184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:38.501115985 +0000 UTC m=+1443.367627226" watchObservedRunningTime="2026-09-09 16:15:38.501850184 +0000 UTC m=+1443.368361405"
Sep 09 16:15:38.644181 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:38.644156    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-5d3e44"
Sep 09 16:15:38.702259 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:38.702228    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7dqx\" (UniqueName: \"kubernetes.io/projected/edb26280-ab77-4822-8ee1-1cb90b963e59-kube-api-access-h7dqx\") pod \"edb26280-ab77-4822-8ee1-1cb90b963e59\" (UID: \"edb26280-ab77-4822-8ee1-1cb90b963e59\") "
Sep 09 16:15:38.704371 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:38.704339    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb26280-ab77-4822-8ee1-1cb90b963e59-kube-api-access-h7dqx" (OuterVolumeSpecName: "kube-api-access-h7dqx") pod "edb26280-ab77-4822-8ee1-1cb90b963e59" (UID: "edb26280-ab77-4822-8ee1-1cb90b963e59"). InnerVolumeSpecName "kube-api-access-h7dqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:15:38.802963 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:38.802876    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7dqx\" (UniqueName: \"kubernetes.io/projected/edb26280-ab77-4822-8ee1-1cb90b963e59-kube-api-access-h7dqx\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:15:39.092734 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:39.092644    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7695-aef458"]
Sep 09 16:15:39.099372 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:39.099339    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7695-aef458"]
Sep 09 16:15:39.494261 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:39.494222    2556 generic.go:356] "Generic (PLEG): container finished" podID="edb26280-ab77-4822-8ee1-1cb90b963e59" containerID="60048c4cb282fa30d1158b05b06da6f7d255c1d24adb576b1db1b2280106b9dc" exitCode=137
Sep 09 16:15:39.494736 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:39.494266    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-5d3e44"
Sep 09 16:15:39.494736 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:39.494314    2556 scope.go:117] "RemoveContainer" containerID="60048c4cb282fa30d1158b05b06da6f7d255c1d24adb576b1db1b2280106b9dc"
Sep 09 16:15:39.494736 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:15:39.494665    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7695-aef458" container="test-curl-7695-aef458"
Sep 09 16:15:39.502807 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:39.502790    2556 scope.go:117] "RemoveContainer" containerID="60048c4cb282fa30d1158b05b06da6f7d255c1d24adb576b1db1b2280106b9dc"
Sep 09 16:15:39.503121 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:15:39.503101    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60048c4cb282fa30d1158b05b06da6f7d255c1d24adb576b1db1b2280106b9dc\": container with ID starting with 60048c4cb282fa30d1158b05b06da6f7d255c1d24adb576b1db1b2280106b9dc not found: ID does not exist" containerID="60048c4cb282fa30d1158b05b06da6f7d255c1d24adb576b1db1b2280106b9dc"
Sep 09 16:15:39.503169 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:39.503130    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60048c4cb282fa30d1158b05b06da6f7d255c1d24adb576b1db1b2280106b9dc"} err="failed to get container status \"60048c4cb282fa30d1158b05b06da6f7d255c1d24adb576b1db1b2280106b9dc\": rpc error: code = NotFound desc = could not find container \"60048c4cb282fa30d1158b05b06da6f7d255c1d24adb576b1db1b2280106b9dc\": container with ID starting with 60048c4cb282fa30d1158b05b06da6f7d255c1d24adb576b1db1b2280106b9dc not found: ID does not exist"
Sep 09 16:15:39.504243 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:15:39.504221    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7695-5d3e44\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="edb26280-ab77-4822-8ee1-1cb90b963e59" pod="opendatahub/test-curl-7695-5d3e44"
Sep 09 16:15:39.680834 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:39.680800    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:39.763627 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:39.763533    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edb26280-ab77-4822-8ee1-1cb90b963e59" path="/var/lib/kubelet/pods/edb26280-ab77-4822-8ee1-1cb90b963e59/volumes"
Sep 09 16:15:39.982266 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:39.982232    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:40.155506 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:40.155387    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:40.500165 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:40.500135    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7695-aef458" podUID="6a8bd5ff-4cab-4fe4-a448-964375cce63b" containerName="test-curl-7695-aef458" containerID="cri-o://979598ad6f7b6dbaa77802b266d0659f70139d0430a17e58440fb60def2ccbca" gracePeriod=2
Sep 09 16:15:40.653425 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:40.653400    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-e6bc38"
Sep 09 16:15:40.718122 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:40.718087    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rc9g\" (UniqueName: \"kubernetes.io/projected/eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c-kube-api-access-9rc9g\") pod \"eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c\" (UID: \"eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c\") "
Sep 09 16:15:40.720115 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:40.720089    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c-kube-api-access-9rc9g" (OuterVolumeSpecName: "kube-api-access-9rc9g") pod "eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c" (UID: "eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c"). InnerVolumeSpecName "kube-api-access-9rc9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:15:40.819364 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:40.819282    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rc9g\" (UniqueName: \"kubernetes.io/projected/eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c-kube-api-access-9rc9g\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:15:41.506301 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:41.506262    2556 generic.go:356] "Generic (PLEG): container finished" podID="eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c" containerID="c088eab9c905cac357ffd562d76000da8acc2cf636483db38997640f72ffe5f9" exitCode=137
Sep 09 16:15:41.506301 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:41.506300    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-e6bc38"
Sep 09 16:15:41.506805 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:41.506361    2556 scope.go:117] "RemoveContainer" containerID="c088eab9c905cac357ffd562d76000da8acc2cf636483db38997640f72ffe5f9"
Sep 09 16:15:41.515540 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:41.515525    2556 scope.go:117] "RemoveContainer" containerID="c088eab9c905cac357ffd562d76000da8acc2cf636483db38997640f72ffe5f9"
Sep 09 16:15:41.515805 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:15:41.515788    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c088eab9c905cac357ffd562d76000da8acc2cf636483db38997640f72ffe5f9\": container with ID starting with c088eab9c905cac357ffd562d76000da8acc2cf636483db38997640f72ffe5f9 not found: ID does not exist" containerID="c088eab9c905cac357ffd562d76000da8acc2cf636483db38997640f72ffe5f9"
Sep 09 16:15:41.515864 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:41.515812    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c088eab9c905cac357ffd562d76000da8acc2cf636483db38997640f72ffe5f9"} err="failed to get container status \"c088eab9c905cac357ffd562d76000da8acc2cf636483db38997640f72ffe5f9\": rpc error: code = NotFound desc = could not find container \"c088eab9c905cac357ffd562d76000da8acc2cf636483db38997640f72ffe5f9\": container with ID starting with c088eab9c905cac357ffd562d76000da8acc2cf636483db38997640f72ffe5f9 not found: ID does not exist"
Sep 09 16:15:41.516494 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:15:41.516478    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7695-e6bc38\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c" pod="opendatahub/test-curl-7695-e6bc38"
Sep 09 16:15:41.764845 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:41.764755    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c" path="/var/lib/kubelet/pods/eb87cf21-f6a4-482b-8c50-e6c91b2ccc1c/volumes"
Sep 09 16:15:42.691170 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:42.691145    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-aef458"
Sep 09 16:15:42.734102 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:42.734066    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dksbn\" (UniqueName: \"kubernetes.io/projected/6a8bd5ff-4cab-4fe4-a448-964375cce63b-kube-api-access-dksbn\") pod \"6a8bd5ff-4cab-4fe4-a448-964375cce63b\" (UID: \"6a8bd5ff-4cab-4fe4-a448-964375cce63b\") "
Sep 09 16:15:42.736080 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:42.736055    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8bd5ff-4cab-4fe4-a448-964375cce63b-kube-api-access-dksbn" (OuterVolumeSpecName: "kube-api-access-dksbn") pod "6a8bd5ff-4cab-4fe4-a448-964375cce63b" (UID: "6a8bd5ff-4cab-4fe4-a448-964375cce63b"). InnerVolumeSpecName "kube-api-access-dksbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:15:42.834925 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:42.834845    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dksbn\" (UniqueName: \"kubernetes.io/projected/6a8bd5ff-4cab-4fe4-a448-964375cce63b-kube-api-access-dksbn\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:15:43.426545 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:43.426505    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:43.518240 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:43.518205    2556 generic.go:356] "Generic (PLEG): container finished" podID="6a8bd5ff-4cab-4fe4-a448-964375cce63b" containerID="979598ad6f7b6dbaa77802b266d0659f70139d0430a17e58440fb60def2ccbca" exitCode=137
Sep 09 16:15:43.518240 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:43.518244    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-aef458"
Sep 09 16:15:43.518496 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:43.518287    2556 scope.go:117] "RemoveContainer" containerID="979598ad6f7b6dbaa77802b266d0659f70139d0430a17e58440fb60def2ccbca"
Sep 09 16:15:43.526456 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:43.526424    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:43.526828 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:43.526813    2556 scope.go:117] "RemoveContainer" containerID="979598ad6f7b6dbaa77802b266d0659f70139d0430a17e58440fb60def2ccbca"
Sep 09 16:15:43.527115 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:15:43.527098    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"979598ad6f7b6dbaa77802b266d0659f70139d0430a17e58440fb60def2ccbca\": container with ID starting with 979598ad6f7b6dbaa77802b266d0659f70139d0430a17e58440fb60def2ccbca not found: ID does not exist" containerID="979598ad6f7b6dbaa77802b266d0659f70139d0430a17e58440fb60def2ccbca"
Sep 09 16:15:43.527174 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:43.527121    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979598ad6f7b6dbaa77802b266d0659f70139d0430a17e58440fb60def2ccbca"} err="failed to get container status \"979598ad6f7b6dbaa77802b266d0659f70139d0430a17e58440fb60def2ccbca\": rpc error: code = NotFound desc = could not find container \"979598ad6f7b6dbaa77802b266d0659f70139d0430a17e58440fb60def2ccbca\": container with ID starting with 979598ad6f7b6dbaa77802b266d0659f70139d0430a17e58440fb60def2ccbca not found: ID does not exist"
Sep 09 16:15:43.528251 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:15:43.528224    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7695-aef458\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="6a8bd5ff-4cab-4fe4-a448-964375cce63b" pod="opendatahub/test-curl-7695-aef458"
Sep 09 16:15:43.763024 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:43.762938    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a8bd5ff-4cab-4fe4-a448-964375cce63b" path="/var/lib/kubelet/pods/6a8bd5ff-4cab-4fe4-a448-964375cce63b/volumes"
Sep 09 16:15:44.618837 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:44.618795    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:44.756359 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:44.756319    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:44.856935 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:44.856900    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:45.172001 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.171965    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:45.277728 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.277693    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:15:45.277985 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.277965    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"]
Sep 09 16:15:45.278296 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.278285    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8bd5ff-4cab-4fe4-a448-964375cce63b" containerName="test-curl-7695-aef458"
Sep 09 16:15:45.278334 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.278298    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8bd5ff-4cab-4fe4-a448-964375cce63b" containerName="test-curl-7695-aef458"
Sep 09 16:15:45.278373 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.278364    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8bd5ff-4cab-4fe4-a448-964375cce63b" containerName="test-curl-7695-aef458"
Sep 09 16:15:45.282576 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.282558    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"
Sep 09 16:15:45.284615 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.284596    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4c4ad334-dockercfg-6bftc\""
Sep 09 16:15:45.284615 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.284609    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4c4ad334\""
Sep 09 16:15:45.288324 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.288298    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"]
Sep 09 16:15:45.308714 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.308686    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"]
Sep 09 16:15:45.311985 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.311960    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"
Sep 09 16:15:45.323249 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.323225    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"]
Sep 09 16:15:45.360654 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.360618    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-442wg\" (UniqueName: \"kubernetes.io/projected/13ee5d0c-4fa5-4687-9cbf-542e3382cdf4-kube-api-access-442wg\") pod \"payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db\" (UID: \"13ee5d0c-4fa5-4687-9cbf-542e3382cdf4\") " pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"
Sep 09 16:15:45.360836 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.360663    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk\" (UID: \"bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"
Sep 09 16:15:45.360836 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.360795    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ee5d0c-4fa5-4687-9cbf-542e3382cdf4-plugins-config-volume\") pod \"payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db\" (UID: \"13ee5d0c-4fa5-4687-9cbf-542e3382cdf4\") " pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"
Sep 09 16:15:45.360914 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.360860    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j884k\" (UniqueName: \"kubernetes.io/projected/bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f-kube-api-access-j884k\") pod \"payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk\" (UID: \"bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"
Sep 09 16:15:45.461892 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.461853    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-442wg\" (UniqueName: \"kubernetes.io/projected/13ee5d0c-4fa5-4687-9cbf-542e3382cdf4-kube-api-access-442wg\") pod \"payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db\" (UID: \"13ee5d0c-4fa5-4687-9cbf-542e3382cdf4\") " pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"
Sep 09 16:15:45.461892 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.461895    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk\" (UID: \"bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"
Sep 09 16:15:45.462152 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.461931    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ee5d0c-4fa5-4687-9cbf-542e3382cdf4-plugins-config-volume\") pod \"payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db\" (UID: \"13ee5d0c-4fa5-4687-9cbf-542e3382cdf4\") " pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"
Sep 09 16:15:45.462152 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.461961    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j884k\" (UniqueName: \"kubernetes.io/projected/bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f-kube-api-access-j884k\") pod \"payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk\" (UID: \"bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"
Sep 09 16:15:45.462643 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.462619    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk\" (UID: \"bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"
Sep 09 16:15:45.462761 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.462711    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ee5d0c-4fa5-4687-9cbf-542e3382cdf4-plugins-config-volume\") pod \"payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db\" (UID: \"13ee5d0c-4fa5-4687-9cbf-542e3382cdf4\") " pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"
Sep 09 16:15:45.470114 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.470084    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j884k\" (UniqueName: \"kubernetes.io/projected/bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f-kube-api-access-j884k\") pod \"payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk\" (UID: \"bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"
Sep 09 16:15:45.470236 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.470216    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-442wg\" (UniqueName: \"kubernetes.io/projected/13ee5d0c-4fa5-4687-9cbf-542e3382cdf4-kube-api-access-442wg\") pod \"payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db\" (UID: \"13ee5d0c-4fa5-4687-9cbf-542e3382cdf4\") " pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"
Sep 09 16:15:45.593209 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.593176    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"
Sep 09 16:15:45.623788 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.623758    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"
Sep 09 16:15:45.867554 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.867454    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"]
Sep 09 16:15:45.870231 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:15:45.870206    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4bfe9f_1e13_4736_98bc_b5a4b4202f0f.slice/crio-422e2fe9eee699ad40f1ca0f149655c54ec69ae4fab275b173c3758cda3bb6c2 WatchSource:0}: Error finding container 422e2fe9eee699ad40f1ca0f149655c54ec69ae4fab275b173c3758cda3bb6c2: Status 404 returned error can't find the container with id 422e2fe9eee699ad40f1ca0f149655c54ec69ae4fab275b173c3758cda3bb6c2
Sep 09 16:15:45.905169 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:45.905146    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"]
Sep 09 16:15:45.907686 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:15:45.907657    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ee5d0c_4fa5_4687_9cbf_542e3382cdf4.slice/crio-31748834f39ffdb5fd8273920798b60923b579f03216e10dab6c0a55d9fe382a WatchSource:0}: Error finding container 31748834f39ffdb5fd8273920798b60923b579f03216e10dab6c0a55d9fe382a: Status 404 returned error can't find the container with id 31748834f39ffdb5fd8273920798b60923b579f03216e10dab6c0a55d9fe382a
Sep 09 16:15:46.536893 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:46.536855    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db" event={"ID":"13ee5d0c-4fa5-4687-9cbf-542e3382cdf4","Type":"ContainerStarted","Data":"462264ecb2ab611c7ded21203cddade8bd6716ff7dd204080062031cd33d9d41"}
Sep 09 16:15:46.537113 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:46.536935    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db" event={"ID":"13ee5d0c-4fa5-4687-9cbf-542e3382cdf4","Type":"ContainerStarted","Data":"31748834f39ffdb5fd8273920798b60923b579f03216e10dab6c0a55d9fe382a"}
Sep 09 16:15:46.537113 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:46.536964    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"
Sep 09 16:15:46.538935 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:46.538909    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk" event={"ID":"bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f","Type":"ContainerStarted","Data":"d01b3d9673f6e0a7ae2761ff823ac39ab358c42695a314d59c664ce9f126f2f3"}
Sep 09 16:15:46.539063 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:46.538960    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk" event={"ID":"bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f","Type":"ContainerStarted","Data":"422e2fe9eee699ad40f1ca0f149655c54ec69ae4fab275b173c3758cda3bb6c2"}
Sep 09 16:15:46.539063 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:46.538987    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"
Sep 09 16:15:46.555209 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:46.555153    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db" podStartSLOduration=1.555133832 podStartE2EDuration="1.555133832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:46.553370209 +0000 UTC m=+1451.419881442" watchObservedRunningTime="2026-09-09 16:15:46.555133832 +0000 UTC m=+1451.421645054"
Sep 09 16:15:46.570683 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:46.570630    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk" podStartSLOduration=1.5706084150000001 podStartE2EDuration="1.570608415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:46.569823228 +0000 UTC m=+1451.436334453" watchObservedRunningTime="2026-09-09 16:15:46.570608415 +0000 UTC m=+1451.437119653"
Sep 09 16:15:57.547424 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:57.547396    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"
Sep 09 16:15:57.547879 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:15:57.547477    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"
Sep 09 16:16:03.050065 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.050028    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"]
Sep 09 16:16:03.053692 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.053669    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.056039 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.056015    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-13d59f\"/\"kube-root-ca.crt\""
Sep 09 16:16:03.057068 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.057046    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-13d59f\"/\"default-dockercfg-898cn\""
Sep 09 16:16:03.057068 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.057071    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-13d59f\"/\"openshift-service-ca.crt\""
Sep 09 16:16:03.057251 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.057112    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-13d59f\"/\"auth-test-model-13d59f-kserve-self-signed-certs\""
Sep 09 16:16:03.065109 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.065084    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"]
Sep 09 16:16:03.082481 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.082431    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:16:03.122985 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.122949    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-tmp-dir\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.123152 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.122994    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-home\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.123152 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.123055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p79fk\" (UniqueName: \"kubernetes.io/projected/5765644f-6857-4b9b-8518-f38ceb64194e-kube-api-access-p79fk\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.123152 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.123115    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-dshm\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.123261 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.123157    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-model-cache\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.123261 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.123173    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5765644f-6857-4b9b-8518-f38ceb64194e-tls-certs\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.224168 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.224129    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-home\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.224354 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.224182    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p79fk\" (UniqueName: \"kubernetes.io/projected/5765644f-6857-4b9b-8518-f38ceb64194e-kube-api-access-p79fk\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.224354 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.224232    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-dshm\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.224354 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.224276    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-model-cache\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.224525 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.224419    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5765644f-6857-4b9b-8518-f38ceb64194e-tls-certs\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.224592 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.224521    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-tmp-dir\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.224661 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.224639    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-home\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.224661 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.224652    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-model-cache\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.224850 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.224829    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-tmp-dir\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.226446 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.226430    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-dshm\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.226878 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.226857    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5765644f-6857-4b9b-8518-f38ceb64194e-tls-certs\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.232702 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.232681    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p79fk\" (UniqueName: \"kubernetes.io/projected/5765644f-6857-4b9b-8518-f38ceb64194e-kube-api-access-p79fk\") pod \"auth-test-model-13d59f-kserve-55d5f46995-sfnkk\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.364162 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.364060    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:03.639305 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:03.639269    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"]
Sep 09 16:16:04.628183 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:04.628141    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk" event={"ID":"5765644f-6857-4b9b-8518-f38ceb64194e","Type":"ContainerStarted","Data":"7c7ef4e9594d27da83fa2eb68e12e80a24cb680dec2d6a81e7b681027033d4ba"}
Sep 09 16:16:04.628707 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:04.628196    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk" event={"ID":"5765644f-6857-4b9b-8518-f38ceb64194e","Type":"ContainerStarted","Data":"3ae2d497ef85b5bd70d78874e687f5dec1336ce88c4405269822bb86d74a48fc"}
Sep 09 16:16:04.659230 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:04.659176    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk" podStartSLOduration=1.659159831 podStartE2EDuration="1.659159831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:16:04.657092908 +0000 UTC m=+1469.523604125" watchObservedRunningTime="2026-09-09 16:16:04.659159831 +0000 UTC m=+1469.525671091"
Sep 09 16:16:08.322249 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:08.322216    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:16:13.364630 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:13.364591    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:13.365049 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:13.364648    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:13.377010 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:13.376988    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:13.531632 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:13.531597    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:16:13.688481 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:13.688434    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:16:26.156690 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:26.156651    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:16:30.818377 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:30.818338    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:16:30.872988 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:30.872950    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:16:35.684286 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:35.684232    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 16:16:35.686564 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:16:35.686541    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 16:17:01.936342 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:01.936245    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:17:12.958275 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:12.958237    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:17:13.069809 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:13.069762    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:17:13.585427 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:13.585390    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:17:20.881056 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:20.881019    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:17:20.984579 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:20.984539    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:17:23.305881 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:23.305847    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:17:29.324641 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.324596    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:17:29.345972 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.345942    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"]
Sep 09 16:17:29.346234 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.346206    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db" podUID="13ee5d0c-4fa5-4687-9cbf-542e3382cdf4" containerName="payload-processing" containerID="cri-o://462264ecb2ab611c7ded21203cddade8bd6716ff7dd204080062031cd33d9d41" gracePeriod=30
Sep 09 16:17:29.354433 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.354404    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"]
Sep 09 16:17:29.354668 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.354625    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk" podUID="bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f" containerName="payload-pre-processing" containerID="cri-o://d01b3d9673f6e0a7ae2761ff823ac39ab358c42695a314d59c664ce9f126f2f3" gracePeriod=30
Sep 09 16:17:29.391144 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.391096    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:17:29.492912 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.492872    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:17:29.556340 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.556300    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:17:29.601885 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.601861    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"
Sep 09 16:17:29.627689 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.627668    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"
Sep 09 16:17:29.690923 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.690897    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f-plugins-config-volume\") pod \"bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f\" (UID: \"bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f\") "
Sep 09 16:17:29.691078 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.690991    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-442wg\" (UniqueName: \"kubernetes.io/projected/13ee5d0c-4fa5-4687-9cbf-542e3382cdf4-kube-api-access-442wg\") pod \"13ee5d0c-4fa5-4687-9cbf-542e3382cdf4\" (UID: \"13ee5d0c-4fa5-4687-9cbf-542e3382cdf4\") "
Sep 09 16:17:29.691078 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.691009    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ee5d0c-4fa5-4687-9cbf-542e3382cdf4-plugins-config-volume\") pod \"13ee5d0c-4fa5-4687-9cbf-542e3382cdf4\" (UID: \"13ee5d0c-4fa5-4687-9cbf-542e3382cdf4\") "
Sep 09 16:17:29.691078 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.691038    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j884k\" (UniqueName: \"kubernetes.io/projected/bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f-kube-api-access-j884k\") pod \"bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f\" (UID: \"bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f\") "
Sep 09 16:17:29.691341 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.691314    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f" (UID: "bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:17:29.691396 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.691373    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ee5d0c-4fa5-4687-9cbf-542e3382cdf4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13ee5d0c-4fa5-4687-9cbf-542e3382cdf4" (UID: "13ee5d0c-4fa5-4687-9cbf-542e3382cdf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:17:29.693129 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.693106    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f-kube-api-access-j884k" (OuterVolumeSpecName: "kube-api-access-j884k") pod "bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f" (UID: "bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f"). InnerVolumeSpecName "kube-api-access-j884k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:17:29.693195 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.693147    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ee5d0c-4fa5-4687-9cbf-542e3382cdf4-kube-api-access-442wg" (OuterVolumeSpecName: "kube-api-access-442wg") pod "13ee5d0c-4fa5-4687-9cbf-542e3382cdf4" (UID: "13ee5d0c-4fa5-4687-9cbf-542e3382cdf4"). InnerVolumeSpecName "kube-api-access-442wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:17:29.792072 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.792043    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-442wg\" (UniqueName: \"kubernetes.io/projected/13ee5d0c-4fa5-4687-9cbf-542e3382cdf4-kube-api-access-442wg\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:17:29.792072 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.792072    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ee5d0c-4fa5-4687-9cbf-542e3382cdf4-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:17:29.792229 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.792085    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j884k\" (UniqueName: \"kubernetes.io/projected/bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f-kube-api-access-j884k\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:17:29.792229 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:29.792098    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:17:30.040750 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.040710    2556 generic.go:356] "Generic (PLEG): container finished" podID="bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f" containerID="d01b3d9673f6e0a7ae2761ff823ac39ab358c42695a314d59c664ce9f126f2f3" exitCode=0
Sep 09 16:17:30.040929 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.040761    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"
Sep 09 16:17:30.040929 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.040800    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk" event={"ID":"bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f","Type":"ContainerDied","Data":"d01b3d9673f6e0a7ae2761ff823ac39ab358c42695a314d59c664ce9f126f2f3"}
Sep 09 16:17:30.040929 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.040852    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk" event={"ID":"bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f","Type":"ContainerDied","Data":"422e2fe9eee699ad40f1ca0f149655c54ec69ae4fab275b173c3758cda3bb6c2"}
Sep 09 16:17:30.040929 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.040880    2556 scope.go:117] "RemoveContainer" containerID="d01b3d9673f6e0a7ae2761ff823ac39ab358c42695a314d59c664ce9f126f2f3"
Sep 09 16:17:30.042410 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.042384    2556 generic.go:356] "Generic (PLEG): container finished" podID="13ee5d0c-4fa5-4687-9cbf-542e3382cdf4" containerID="462264ecb2ab611c7ded21203cddade8bd6716ff7dd204080062031cd33d9d41" exitCode=0
Sep 09 16:17:30.042597 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.042421    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"
Sep 09 16:17:30.042597 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.042439    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db" event={"ID":"13ee5d0c-4fa5-4687-9cbf-542e3382cdf4","Type":"ContainerDied","Data":"462264ecb2ab611c7ded21203cddade8bd6716ff7dd204080062031cd33d9d41"}
Sep 09 16:17:30.042597 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.042503    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db" event={"ID":"13ee5d0c-4fa5-4687-9cbf-542e3382cdf4","Type":"ContainerDied","Data":"31748834f39ffdb5fd8273920798b60923b579f03216e10dab6c0a55d9fe382a"}
Sep 09 16:17:30.049719 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.049703    2556 scope.go:117] "RemoveContainer" containerID="d01b3d9673f6e0a7ae2761ff823ac39ab358c42695a314d59c664ce9f126f2f3"
Sep 09 16:17:30.049990 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:17:30.049972    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d01b3d9673f6e0a7ae2761ff823ac39ab358c42695a314d59c664ce9f126f2f3\": container with ID starting with d01b3d9673f6e0a7ae2761ff823ac39ab358c42695a314d59c664ce9f126f2f3 not found: ID does not exist" containerID="d01b3d9673f6e0a7ae2761ff823ac39ab358c42695a314d59c664ce9f126f2f3"
Sep 09 16:17:30.050060 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.050002    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01b3d9673f6e0a7ae2761ff823ac39ab358c42695a314d59c664ce9f126f2f3"} err="failed to get container status \"d01b3d9673f6e0a7ae2761ff823ac39ab358c42695a314d59c664ce9f126f2f3\": rpc error: code = NotFound desc = could not find container \"d01b3d9673f6e0a7ae2761ff823ac39ab358c42695a314d59c664ce9f126f2f3\": container with ID starting with d01b3d9673f6e0a7ae2761ff823ac39ab358c42695a314d59c664ce9f126f2f3 not found: ID does not exist"
Sep 09 16:17:30.050060 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.050026    2556 scope.go:117] "RemoveContainer" containerID="462264ecb2ab611c7ded21203cddade8bd6716ff7dd204080062031cd33d9d41"
Sep 09 16:17:30.057759 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.057739    2556 scope.go:117] "RemoveContainer" containerID="462264ecb2ab611c7ded21203cddade8bd6716ff7dd204080062031cd33d9d41"
Sep 09 16:17:30.058041 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:17:30.058022    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462264ecb2ab611c7ded21203cddade8bd6716ff7dd204080062031cd33d9d41\": container with ID starting with 462264ecb2ab611c7ded21203cddade8bd6716ff7dd204080062031cd33d9d41 not found: ID does not exist" containerID="462264ecb2ab611c7ded21203cddade8bd6716ff7dd204080062031cd33d9d41"
Sep 09 16:17:30.058104 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.058051    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462264ecb2ab611c7ded21203cddade8bd6716ff7dd204080062031cd33d9d41"} err="failed to get container status \"462264ecb2ab611c7ded21203cddade8bd6716ff7dd204080062031cd33d9d41\": rpc error: code = NotFound desc = could not find container \"462264ecb2ab611c7ded21203cddade8bd6716ff7dd204080062031cd33d9d41\": container with ID starting with 462264ecb2ab611c7ded21203cddade8bd6716ff7dd204080062031cd33d9d41 not found: ID does not exist"
Sep 09 16:17:30.068875 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.068855    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"]
Sep 09 16:17:30.073023 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.073003    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c4ad334-dc6fbf878-ttfsk"]
Sep 09 16:17:30.091272 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.091250    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"]
Sep 09 16:17:30.095688 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:30.095669    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4c4ad334-69558c697c-sr4db"]
Sep 09 16:17:31.763156 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:31.763123    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13ee5d0c-4fa5-4687-9cbf-542e3382cdf4" path="/var/lib/kubelet/pods/13ee5d0c-4fa5-4687-9cbf-542e3382cdf4/volumes"
Sep 09 16:17:31.763559 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:31.763488    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f" path="/var/lib/kubelet/pods/bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f/volumes"
Sep 09 16:17:32.707077 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:32.707043    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:17:37.748512 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:37.748448    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:17:40.144413 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:17:40.144382    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:07.188491 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:07.188419    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:13.833480 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:13.833423    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:15.020660 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:15.020624    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:16.012312 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:16.012281    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7698-18715a"]
Sep 09 16:18:16.012735 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:16.012718    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ee5d0c-4fa5-4687-9cbf-542e3382cdf4" containerName="payload-processing"
Sep 09 16:18:16.012735 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:16.012736    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ee5d0c-4fa5-4687-9cbf-542e3382cdf4" containerName="payload-processing"
Sep 09 16:18:16.012852 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:16.012748    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f" containerName="payload-pre-processing"
Sep 09 16:18:16.012852 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:16.012754    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f" containerName="payload-pre-processing"
Sep 09 16:18:16.012852 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:16.012818    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="13ee5d0c-4fa5-4687-9cbf-542e3382cdf4" containerName="payload-processing"
Sep 09 16:18:16.012852 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:16.012828    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf4bfe9f-1e13-4736-98bc-b5a4b4202f0f" containerName="payload-pre-processing"
Sep 09 16:18:16.015608 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:16.015593    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7698-18715a"
Sep 09 16:18:16.018511 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:16.018487    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lp4n9\""
Sep 09 16:18:16.020226 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:16.020204    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7698-18715a"]
Sep 09 16:18:16.070600 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:16.070569    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljqjw\" (UniqueName: \"kubernetes.io/projected/1538d7a0-9416-4b5c-80e2-8d1ea65998ac-kube-api-access-ljqjw\") pod \"test-curl-7698-18715a\" (UID: \"1538d7a0-9416-4b5c-80e2-8d1ea65998ac\") " pod="opendatahub/test-curl-7698-18715a"
Sep 09 16:18:16.171640 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:16.171600    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljqjw\" (UniqueName: \"kubernetes.io/projected/1538d7a0-9416-4b5c-80e2-8d1ea65998ac-kube-api-access-ljqjw\") pod \"test-curl-7698-18715a\" (UID: \"1538d7a0-9416-4b5c-80e2-8d1ea65998ac\") " pod="opendatahub/test-curl-7698-18715a"
Sep 09 16:18:16.180846 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:16.180821    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljqjw\" (UniqueName: \"kubernetes.io/projected/1538d7a0-9416-4b5c-80e2-8d1ea65998ac-kube-api-access-ljqjw\") pod \"test-curl-7698-18715a\" (UID: \"1538d7a0-9416-4b5c-80e2-8d1ea65998ac\") " pod="opendatahub/test-curl-7698-18715a"
Sep 09 16:18:16.326534 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:16.326401    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7698-18715a"
Sep 09 16:18:16.595623 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:16.595550    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7698-18715a"]
Sep 09 16:18:16.597869 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:18:16.597836    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1538d7a0_9416_4b5c_80e2_8d1ea65998ac.slice/crio-a8f44e457ebd7a6a117fa4d628d65f7f78742e2e00c073ed546304dc5dd441cf WatchSource:0}: Error finding container a8f44e457ebd7a6a117fa4d628d65f7f78742e2e00c073ed546304dc5dd441cf: Status 404 returned error can't find the container with id a8f44e457ebd7a6a117fa4d628d65f7f78742e2e00c073ed546304dc5dd441cf
Sep 09 16:18:17.262929 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:17.262891    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7698-18715a" event={"ID":"1538d7a0-9416-4b5c-80e2-8d1ea65998ac","Type":"ContainerStarted","Data":"4fffea2ca30dde070341ca0651f791976ffe6529b91b8f45c548f26050958b70"}
Sep 09 16:18:17.263425 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:17.262938    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7698-18715a" event={"ID":"1538d7a0-9416-4b5c-80e2-8d1ea65998ac","Type":"ContainerStarted","Data":"a8f44e457ebd7a6a117fa4d628d65f7f78742e2e00c073ed546304dc5dd441cf"}
Sep 09 16:18:17.276977 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:17.276929    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7698-18715a" podStartSLOduration=2.276914145 podStartE2EDuration="2.276914145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:18:17.275657903 +0000 UTC m=+1602.142169123" watchObservedRunningTime="2026-09-09 16:18:17.276914145 +0000 UTC m=+1602.143425364"
Sep 09 16:18:17.853732 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:17.853695    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7698-18715a"]
Sep 09 16:18:17.857806 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:17.857776    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7698-18715a"]
Sep 09 16:18:18.267988 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:18:18.267962    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7698-18715a" container="test-curl-7698-18715a"
Sep 09 16:18:19.205485 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:19.205432    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:19.272409 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:19.272367    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7698-18715a" podUID="1538d7a0-9416-4b5c-80e2-8d1ea65998ac" containerName="test-curl-7698-18715a" containerID="cri-o://4fffea2ca30dde070341ca0651f791976ffe6529b91b8f45c548f26050958b70" gracePeriod=2
Sep 09 16:18:19.275412 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:18:19.275385    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7698-18715a\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="1538d7a0-9416-4b5c-80e2-8d1ea65998ac" pod="opendatahub/test-curl-7698-18715a"
Sep 09 16:18:20.139191 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:20.139149    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"]
Sep 09 16:18:20.156442 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:20.156394    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:20.606459 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:20.606418    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:20.705945 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:20.705913    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:20.920334 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:20.920225    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:21.444145 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:21.444120    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7698-18715a"
Sep 09 16:18:21.517131 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:21.517098    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljqjw\" (UniqueName: \"kubernetes.io/projected/1538d7a0-9416-4b5c-80e2-8d1ea65998ac-kube-api-access-ljqjw\") pod \"1538d7a0-9416-4b5c-80e2-8d1ea65998ac\" (UID: \"1538d7a0-9416-4b5c-80e2-8d1ea65998ac\") "
Sep 09 16:18:21.519114 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:21.519086    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1538d7a0-9416-4b5c-80e2-8d1ea65998ac-kube-api-access-ljqjw" (OuterVolumeSpecName: "kube-api-access-ljqjw") pod "1538d7a0-9416-4b5c-80e2-8d1ea65998ac" (UID: "1538d7a0-9416-4b5c-80e2-8d1ea65998ac"). InnerVolumeSpecName "kube-api-access-ljqjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:18:21.617890 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:21.617801    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljqjw\" (UniqueName: \"kubernetes.io/projected/1538d7a0-9416-4b5c-80e2-8d1ea65998ac-kube-api-access-ljqjw\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:18:21.763396 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:21.763362    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1538d7a0-9416-4b5c-80e2-8d1ea65998ac" path="/var/lib/kubelet/pods/1538d7a0-9416-4b5c-80e2-8d1ea65998ac/volumes"
Sep 09 16:18:22.133281 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:22.133252    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:22.231562 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:22.231520    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:22.287023 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:22.286990    2556 generic.go:356] "Generic (PLEG): container finished" podID="1538d7a0-9416-4b5c-80e2-8d1ea65998ac" containerID="4fffea2ca30dde070341ca0651f791976ffe6529b91b8f45c548f26050958b70" exitCode=137
Sep 09 16:18:22.287216 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:22.287037    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7698-18715a"
Sep 09 16:18:22.287216 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:22.287053    2556 scope.go:117] "RemoveContainer" containerID="4fffea2ca30dde070341ca0651f791976ffe6529b91b8f45c548f26050958b70"
Sep 09 16:18:22.295733 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:22.295714    2556 scope.go:117] "RemoveContainer" containerID="4fffea2ca30dde070341ca0651f791976ffe6529b91b8f45c548f26050958b70"
Sep 09 16:18:22.296012 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:18:22.295994    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fffea2ca30dde070341ca0651f791976ffe6529b91b8f45c548f26050958b70\": container with ID starting with 4fffea2ca30dde070341ca0651f791976ffe6529b91b8f45c548f26050958b70 not found: ID does not exist" containerID="4fffea2ca30dde070341ca0651f791976ffe6529b91b8f45c548f26050958b70"
Sep 09 16:18:22.296060 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:22.296022    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fffea2ca30dde070341ca0651f791976ffe6529b91b8f45c548f26050958b70"} err="failed to get container status \"4fffea2ca30dde070341ca0651f791976ffe6529b91b8f45c548f26050958b70\": rpc error: code = NotFound desc = could not find container \"4fffea2ca30dde070341ca0651f791976ffe6529b91b8f45c548f26050958b70\": container with ID starting with 4fffea2ca30dde070341ca0651f791976ffe6529b91b8f45c548f26050958b70 not found: ID does not exist"
Sep 09 16:18:22.332137 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:22.332103    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:22.956767 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:22.956738    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:25.299279 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:25.299238    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:25.398915 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:25.398875    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:30.386537 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:30.386426    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:30.486736 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:30.486702    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:33.269148 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:33.269098    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:35.159637 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.159602    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk" podUID="5765644f-6857-4b9b-8518-f38ceb64194e" containerName="main" containerID="cri-o://7c7ef4e9594d27da83fa2eb68e12e80a24cb680dec2d6a81e7b681027033d4ba" gracePeriod=45
Sep 09 16:18:35.352617 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.352588    2556 generic.go:356] "Generic (PLEG): container finished" podID="5765644f-6857-4b9b-8518-f38ceb64194e" containerID="7c7ef4e9594d27da83fa2eb68e12e80a24cb680dec2d6a81e7b681027033d4ba" exitCode=0
Sep 09 16:18:35.352762 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.352638    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk" event={"ID":"5765644f-6857-4b9b-8518-f38ceb64194e","Type":"ContainerDied","Data":"7c7ef4e9594d27da83fa2eb68e12e80a24cb680dec2d6a81e7b681027033d4ba"}
Sep 09 16:18:35.408648 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.408623    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:18:35.542478 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.542439    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-home\") pod \"5765644f-6857-4b9b-8518-f38ceb64194e\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") "
Sep 09 16:18:35.542671 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.542512    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5765644f-6857-4b9b-8518-f38ceb64194e-tls-certs\") pod \"5765644f-6857-4b9b-8518-f38ceb64194e\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") "
Sep 09 16:18:35.542671 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.542543    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-tmp-dir\") pod \"5765644f-6857-4b9b-8518-f38ceb64194e\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") "
Sep 09 16:18:35.542671 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.542603    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-dshm\") pod \"5765644f-6857-4b9b-8518-f38ceb64194e\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") "
Sep 09 16:18:35.542671 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.542635    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p79fk\" (UniqueName: \"kubernetes.io/projected/5765644f-6857-4b9b-8518-f38ceb64194e-kube-api-access-p79fk\") pod \"5765644f-6857-4b9b-8518-f38ceb64194e\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") "
Sep 09 16:18:35.542671 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.542651    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-model-cache\") pod \"5765644f-6857-4b9b-8518-f38ceb64194e\" (UID: \"5765644f-6857-4b9b-8518-f38ceb64194e\") "
Sep 09 16:18:35.542948 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.542709    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-home" (OuterVolumeSpecName: "home") pod "5765644f-6857-4b9b-8518-f38ceb64194e" (UID: "5765644f-6857-4b9b-8518-f38ceb64194e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:18:35.542948 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.542882    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-home\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:18:35.542948 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.542894    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5765644f-6857-4b9b-8518-f38ceb64194e" (UID: "5765644f-6857-4b9b-8518-f38ceb64194e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:18:35.542948 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.542911    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-model-cache" (OuterVolumeSpecName: "model-cache") pod "5765644f-6857-4b9b-8518-f38ceb64194e" (UID: "5765644f-6857-4b9b-8518-f38ceb64194e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:18:35.544846 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.544823    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5765644f-6857-4b9b-8518-f38ceb64194e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5765644f-6857-4b9b-8518-f38ceb64194e" (UID: "5765644f-6857-4b9b-8518-f38ceb64194e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:18:35.544937 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.544842    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5765644f-6857-4b9b-8518-f38ceb64194e-kube-api-access-p79fk" (OuterVolumeSpecName: "kube-api-access-p79fk") pod "5765644f-6857-4b9b-8518-f38ceb64194e" (UID: "5765644f-6857-4b9b-8518-f38ceb64194e"). InnerVolumeSpecName "kube-api-access-p79fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:18:35.544937 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.544885    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-dshm" (OuterVolumeSpecName: "dshm") pod "5765644f-6857-4b9b-8518-f38ceb64194e" (UID: "5765644f-6857-4b9b-8518-f38ceb64194e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:18:35.644069 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.644023    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-model-cache\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:18:35.644249 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.644088    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5765644f-6857-4b9b-8518-f38ceb64194e-tls-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:18:35.644249 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.644100    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-tmp-dir\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:18:35.644249 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.644111    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5765644f-6857-4b9b-8518-f38ceb64194e-dshm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:18:35.644249 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:35.644123    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p79fk\" (UniqueName: \"kubernetes.io/projected/5765644f-6857-4b9b-8518-f38ceb64194e-kube-api-access-p79fk\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:18:36.358649 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:36.358609    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk" event={"ID":"5765644f-6857-4b9b-8518-f38ceb64194e","Type":"ContainerDied","Data":"3ae2d497ef85b5bd70d78874e687f5dec1336ce88c4405269822bb86d74a48fc"}
Sep 09 16:18:36.359138 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:36.358662    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"
Sep 09 16:18:36.359138 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:36.358669    2556 scope.go:117] "RemoveContainer" containerID="7c7ef4e9594d27da83fa2eb68e12e80a24cb680dec2d6a81e7b681027033d4ba"
Sep 09 16:18:36.374739 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:36.374704    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"]
Sep 09 16:18:36.376272 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:36.376251    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/auth-test-model-13d59f-kserve-55d5f46995-sfnkk"]
Sep 09 16:18:37.098835 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:37.098788    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:37.204371 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:37.204340    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:37.299862 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:37.299820    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:37.763044 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:37.763010    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5765644f-6857-4b9b-8518-f38ceb64194e" path="/var/lib/kubelet/pods/5765644f-6857-4b9b-8518-f38ceb64194e/volumes"
Sep 09 16:18:41.361191 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:41.361157    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:47.831771 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:47.831731    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:18:48.063889 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:18:48.063854    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:09.024695 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:09.024646    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:14.793098 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:14.793055    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:14.899070 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:14.899031    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:15.201888 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:15.201852    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:15.502785 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:15.502696    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:15.606171 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:15.606136    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:18.763822 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:18.763784    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:18.854295 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:18.854264    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:19.935518 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:19.935482    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:20.050888 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.050848    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:20.326836 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.326753    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:20.522416 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.522373    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:20.610854 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.610766    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:20.614893 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.614868    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"]
Sep 09 16:19:20.615248 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.615234    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1538d7a0-9416-4b5c-80e2-8d1ea65998ac" containerName="test-curl-7698-18715a"
Sep 09 16:19:20.615292 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.615250    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1538d7a0-9416-4b5c-80e2-8d1ea65998ac" containerName="test-curl-7698-18715a"
Sep 09 16:19:20.615292 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.615261    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5765644f-6857-4b9b-8518-f38ceb64194e" containerName="main"
Sep 09 16:19:20.615292 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.615267    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5765644f-6857-4b9b-8518-f38ceb64194e" containerName="main"
Sep 09 16:19:20.615387 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.615333    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5765644f-6857-4b9b-8518-f38ceb64194e" containerName="main"
Sep 09 16:19:20.615387 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.615343    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1538d7a0-9416-4b5c-80e2-8d1ea65998ac" containerName="test-curl-7698-18715a"
Sep 09 16:19:20.618276 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.618261    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"
Sep 09 16:19:20.621346 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.621327    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-94cf14\""
Sep 09 16:19:20.634032 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.634009    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"]
Sep 09 16:19:20.656454 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.656423    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"]
Sep 09 16:19:20.659738 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.659721    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"
Sep 09 16:19:20.663300 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.663278    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-94cf14-dockercfg-5nv64\""
Sep 09 16:19:20.671034 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.671010    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"]
Sep 09 16:19:20.713445 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.713408    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08ede8c-1356-42eb-856f-b3264fd620b6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv\" (UID: \"b08ede8c-1356-42eb-856f-b3264fd620b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"
Sep 09 16:19:20.713445 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.713447    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a779c8-7c00-4d90-a1ac-871de28e796c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc\" (UID: \"99a779c8-7c00-4d90-a1ac-871de28e796c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"
Sep 09 16:19:20.713673 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.713499    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tknfr\" (UniqueName: \"kubernetes.io/projected/b08ede8c-1356-42eb-856f-b3264fd620b6-kube-api-access-tknfr\") pod \"payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv\" (UID: \"b08ede8c-1356-42eb-856f-b3264fd620b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"
Sep 09 16:19:20.713673 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.713610    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgwl9\" (UniqueName: \"kubernetes.io/projected/99a779c8-7c00-4d90-a1ac-871de28e796c-kube-api-access-lgwl9\") pod \"payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc\" (UID: \"99a779c8-7c00-4d90-a1ac-871de28e796c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"
Sep 09 16:19:20.814159 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.814124    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwl9\" (UniqueName: \"kubernetes.io/projected/99a779c8-7c00-4d90-a1ac-871de28e796c-kube-api-access-lgwl9\") pod \"payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc\" (UID: \"99a779c8-7c00-4d90-a1ac-871de28e796c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"
Sep 09 16:19:20.814364 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.814186    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08ede8c-1356-42eb-856f-b3264fd620b6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv\" (UID: \"b08ede8c-1356-42eb-856f-b3264fd620b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"
Sep 09 16:19:20.814364 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.814229    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a779c8-7c00-4d90-a1ac-871de28e796c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc\" (UID: \"99a779c8-7c00-4d90-a1ac-871de28e796c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"
Sep 09 16:19:20.814364 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.814266    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknfr\" (UniqueName: \"kubernetes.io/projected/b08ede8c-1356-42eb-856f-b3264fd620b6-kube-api-access-tknfr\") pod \"payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv\" (UID: \"b08ede8c-1356-42eb-856f-b3264fd620b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"
Sep 09 16:19:20.815038 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.815013    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08ede8c-1356-42eb-856f-b3264fd620b6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv\" (UID: \"b08ede8c-1356-42eb-856f-b3264fd620b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"
Sep 09 16:19:20.815135 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.815119    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a779c8-7c00-4d90-a1ac-871de28e796c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc\" (UID: \"99a779c8-7c00-4d90-a1ac-871de28e796c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"
Sep 09 16:19:20.828032 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.827998    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tknfr\" (UniqueName: \"kubernetes.io/projected/b08ede8c-1356-42eb-856f-b3264fd620b6-kube-api-access-tknfr\") pod \"payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv\" (UID: \"b08ede8c-1356-42eb-856f-b3264fd620b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"
Sep 09 16:19:20.828416 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.828396    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwl9\" (UniqueName: \"kubernetes.io/projected/99a779c8-7c00-4d90-a1ac-871de28e796c-kube-api-access-lgwl9\") pod \"payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc\" (UID: \"99a779c8-7c00-4d90-a1ac-871de28e796c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"
Sep 09 16:19:20.928296 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.928268    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"
Sep 09 16:19:20.970188 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:20.970156    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"
Sep 09 16:19:21.201117 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:21.201068    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"]
Sep 09 16:19:21.203491 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:19:21.203446    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a779c8_7c00_4d90_a1ac_871de28e796c.slice/crio-ae0bb62d160c201694a2b47b28a7744e0c8e8f7c6d370f10aaa0daa8754613ec WatchSource:0}: Error finding container ae0bb62d160c201694a2b47b28a7744e0c8e8f7c6d370f10aaa0daa8754613ec: Status 404 returned error can't find the container with id ae0bb62d160c201694a2b47b28a7744e0c8e8f7c6d370f10aaa0daa8754613ec
Sep 09 16:19:21.258217 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:21.258187    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"]
Sep 09 16:19:21.261424 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:19:21.261394    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08ede8c_1356_42eb_856f_b3264fd620b6.slice/crio-dccfed70cb3c1b1199830cb8ebbbfbb510b2263160921c938dc35b89e944138e WatchSource:0}: Error finding container dccfed70cb3c1b1199830cb8ebbbfbb510b2263160921c938dc35b89e944138e: Status 404 returned error can't find the container with id dccfed70cb3c1b1199830cb8ebbbfbb510b2263160921c938dc35b89e944138e
Sep 09 16:19:21.571818 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:21.571710    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv" event={"ID":"b08ede8c-1356-42eb-856f-b3264fd620b6","Type":"ContainerStarted","Data":"94b68a31e8a15a0aa0a6ded0ec9ecc1140324d5b546b0cbd8673a299e7642755"}
Sep 09 16:19:21.571818 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:21.571779    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv" event={"ID":"b08ede8c-1356-42eb-856f-b3264fd620b6","Type":"ContainerStarted","Data":"dccfed70cb3c1b1199830cb8ebbbfbb510b2263160921c938dc35b89e944138e"}
Sep 09 16:19:21.572060 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:21.571819    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"
Sep 09 16:19:21.573522 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:21.573495    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc" event={"ID":"99a779c8-7c00-4d90-a1ac-871de28e796c","Type":"ContainerStarted","Data":"6a801dfba907e7bf5f3fed36650f743ac13b8ede74aaa01c995d3fea24976e13"}
Sep 09 16:19:21.573637 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:21.573543    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc" event={"ID":"99a779c8-7c00-4d90-a1ac-871de28e796c","Type":"ContainerStarted","Data":"ae0bb62d160c201694a2b47b28a7744e0c8e8f7c6d370f10aaa0daa8754613ec"}
Sep 09 16:19:21.573637 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:21.573571    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"
Sep 09 16:19:21.592123 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:21.592065    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv" podStartSLOduration=1.59204985 podStartE2EDuration="1.59204985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:19:21.590182494 +0000 UTC m=+1666.456693714" watchObservedRunningTime="2026-09-09 16:19:21.59204985 +0000 UTC m=+1666.458561070"
Sep 09 16:19:21.614003 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:21.613952    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc" podStartSLOduration=1.613938735 podStartE2EDuration="1.613938735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:19:21.612258764 +0000 UTC m=+1666.478770206" watchObservedRunningTime="2026-09-09 16:19:21.613938735 +0000 UTC m=+1666.480449992"
Sep 09 16:19:26.569023 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:26.568987    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:30.792016 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:30.791969    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:32.580026 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:32.579992    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"
Sep 09 16:19:32.580548 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:32.580077    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"
Sep 09 16:19:35.830523 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:35.830487    2556 scope.go:117] "RemoveContainer" containerID="e72540393809f9599b16e4ed880381a476e4a8873090510d4300269a4d897ed7"
Sep 09 16:19:41.071538 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:41.071488    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:43.701780 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:43.701746    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:44.145438 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:44.145345    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:44.445978 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:44.445940    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:44.922933 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:44.922845    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:47.886758 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:47.886720    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:47.978315 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:47.978285    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:49.099900 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:49.099866    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:49.199415 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:49.199379    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:49.326757 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:49.326711    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:49.699913 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:49.699878    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:49.814247 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:49.814210    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:19:49.861527 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:49.861493    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"]
Sep 09 16:19:49.864762 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:49.864745    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"
Sep 09 16:19:49.870035 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:49.870012    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-07cc17\""
Sep 09 16:19:49.883613 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:49.883589    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"]
Sep 09 16:19:49.933596 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:49.933564    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"]
Sep 09 16:19:49.937102 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:49.937084    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"
Sep 09 16:19:49.940033 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:49.940005    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-07cc17-dockercfg-6g6fq\""
Sep 09 16:19:49.948979 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:49.948957    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"]
Sep 09 16:19:49.984883 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:49.984802    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxd89\" (UniqueName: \"kubernetes.io/projected/ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4-kube-api-access-lxd89\") pod \"payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6\" (UID: \"ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4\") " pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"
Sep 09 16:19:49.984883 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:49.984877    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6\" (UID: \"ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4\") " pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"
Sep 09 16:19:50.086167 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.086128    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6\" (UID: \"ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4\") " pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"
Sep 09 16:19:50.086341 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.086184    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt\" (UID: \"4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"
Sep 09 16:19:50.086341 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.086232    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxd89\" (UniqueName: \"kubernetes.io/projected/ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4-kube-api-access-lxd89\") pod \"payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6\" (UID: \"ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4\") " pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"
Sep 09 16:19:50.086341 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.086286    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2kn7\" (UniqueName: \"kubernetes.io/projected/4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2-kube-api-access-d2kn7\") pod \"payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt\" (UID: \"4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"
Sep 09 16:19:50.086911 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.086890    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6\" (UID: \"ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4\") " pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"
Sep 09 16:19:50.095175 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.095129    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxd89\" (UniqueName: \"kubernetes.io/projected/ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4-kube-api-access-lxd89\") pod \"payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6\" (UID: \"ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4\") " pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"
Sep 09 16:19:50.174366 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.174328    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"
Sep 09 16:19:50.186743 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.186714    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2kn7\" (UniqueName: \"kubernetes.io/projected/4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2-kube-api-access-d2kn7\") pod \"payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt\" (UID: \"4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"
Sep 09 16:19:50.186861 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.186794    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt\" (UID: \"4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"
Sep 09 16:19:50.187452 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.187432    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt\" (UID: \"4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"
Sep 09 16:19:50.195524 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.195504    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2kn7\" (UniqueName: \"kubernetes.io/projected/4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2-kube-api-access-d2kn7\") pod \"payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt\" (UID: \"4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"
Sep 09 16:19:50.247775 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.247695    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"
Sep 09 16:19:50.444498 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.444458    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"]
Sep 09 16:19:50.445974 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:19:50.445945    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8aca0a_f93a_43cb_8a13_f0bf15b99ef4.slice/crio-89800f2f5b5e8c933d6e3df706805a3f7c5d03afc2322be1785d3a9a3369d34e WatchSource:0}: Error finding container 89800f2f5b5e8c933d6e3df706805a3f7c5d03afc2322be1785d3a9a3369d34e: Status 404 returned error can't find the container with id 89800f2f5b5e8c933d6e3df706805a3f7c5d03afc2322be1785d3a9a3369d34e
Sep 09 16:19:50.521762 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.521661    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"]
Sep 09 16:19:50.523739 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:19:50.523712    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1ef883_0ff8_4ca7_9ae5_8a02c03c2ca2.slice/crio-0a52d07aa6cf3b97cc9de497066c3846497a2cf3a91342d9c0bdc3b2a1f87d15 WatchSource:0}: Error finding container 0a52d07aa6cf3b97cc9de497066c3846497a2cf3a91342d9c0bdc3b2a1f87d15: Status 404 returned error can't find the container with id 0a52d07aa6cf3b97cc9de497066c3846497a2cf3a91342d9c0bdc3b2a1f87d15
Sep 09 16:19:50.723173 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.723135    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6" event={"ID":"ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4","Type":"ContainerStarted","Data":"7efccb1175ccd98102203b98200b82d294d7fd04b8272d43987470bbc7f4d0a2"}
Sep 09 16:19:50.723408 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.723381    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"
Sep 09 16:19:50.723559 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.723430    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6" event={"ID":"ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4","Type":"ContainerStarted","Data":"89800f2f5b5e8c933d6e3df706805a3f7c5d03afc2322be1785d3a9a3369d34e"}
Sep 09 16:19:50.724976 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.724948    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt" event={"ID":"4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2","Type":"ContainerStarted","Data":"5b4abb14684fd11105e68a803ac2317774d4d475e2d2b8d7c7ebc98ef2da724e"}
Sep 09 16:19:50.725084 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.724996    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt" event={"ID":"4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2","Type":"ContainerStarted","Data":"0a52d07aa6cf3b97cc9de497066c3846497a2cf3a91342d9c0bdc3b2a1f87d15"}
Sep 09 16:19:50.725084 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.725037    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"
Sep 09 16:19:50.740444 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.740394    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6" podStartSLOduration=1.740374889 podStartE2EDuration="1.740374889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:19:50.738269127 +0000 UTC m=+1695.604780367" watchObservedRunningTime="2026-09-09 16:19:50.740374889 +0000 UTC m=+1695.606886111"
Sep 09 16:19:50.755939 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:50.755837    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt" podStartSLOduration=1.755819683 podStartE2EDuration="1.755819683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:19:50.755057573 +0000 UTC m=+1695.621568794" watchObservedRunningTime="2026-09-09 16:19:50.755819683 +0000 UTC m=+1695.622330906"
Sep 09 16:19:59.044154 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:19:59.044065    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:00.373517 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:00.373478    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:01.731905 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:01.731874    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"
Sep 09 16:20:01.732300 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:01.731942    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"
Sep 09 16:20:05.043157 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:05.043118    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7698-216eee"]
Sep 09 16:20:05.047924 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:05.047906    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7698-216eee"
Sep 09 16:20:05.050745 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:05.050725    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lp4n9\""
Sep 09 16:20:05.058391 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:05.058365    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7698-216eee"]
Sep 09 16:20:05.100770 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:05.100729    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcnr7\" (UniqueName: \"kubernetes.io/projected/7785ee09-c3da-44fa-b208-88de0d5416ee-kube-api-access-gcnr7\") pod \"test-curl-7698-216eee\" (UID: \"7785ee09-c3da-44fa-b208-88de0d5416ee\") " pod="opendatahub/test-curl-7698-216eee"
Sep 09 16:20:05.201545 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:05.201504    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnr7\" (UniqueName: \"kubernetes.io/projected/7785ee09-c3da-44fa-b208-88de0d5416ee-kube-api-access-gcnr7\") pod \"test-curl-7698-216eee\" (UID: \"7785ee09-c3da-44fa-b208-88de0d5416ee\") " pod="opendatahub/test-curl-7698-216eee"
Sep 09 16:20:05.209140 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:05.209112    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcnr7\" (UniqueName: \"kubernetes.io/projected/7785ee09-c3da-44fa-b208-88de0d5416ee-kube-api-access-gcnr7\") pod \"test-curl-7698-216eee\" (UID: \"7785ee09-c3da-44fa-b208-88de0d5416ee\") " pod="opendatahub/test-curl-7698-216eee"
Sep 09 16:20:05.356947 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:05.356853    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7698-216eee"
Sep 09 16:20:05.624746 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:05.624720    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7698-216eee"]
Sep 09 16:20:05.626659 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:20:05.626617    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7785ee09_c3da_44fa_b208_88de0d5416ee.slice/crio-40ed817e774f97068bd4645833e92620dac8355ff4bb88686884024024a35ee0 WatchSource:0}: Error finding container 40ed817e774f97068bd4645833e92620dac8355ff4bb88686884024024a35ee0: Status 404 returned error can't find the container with id 40ed817e774f97068bd4645833e92620dac8355ff4bb88686884024024a35ee0
Sep 09 16:20:05.799430 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:05.799392    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7698-216eee" event={"ID":"7785ee09-c3da-44fa-b208-88de0d5416ee","Type":"ContainerStarted","Data":"44ef574207c0a290f6905351b0f17a31306ecdb9c62bc6eb3c5209b7a1d6a9be"}
Sep 09 16:20:05.799625 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:05.799444    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7698-216eee" event={"ID":"7785ee09-c3da-44fa-b208-88de0d5416ee","Type":"ContainerStarted","Data":"40ed817e774f97068bd4645833e92620dac8355ff4bb88686884024024a35ee0"}
Sep 09 16:20:05.812532 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:05.812457    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7698-216eee" podStartSLOduration=0.812440201 podStartE2EDuration="812.440201ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:20:05.812106864 +0000 UTC m=+1710.678618107" watchObservedRunningTime="2026-09-09 16:20:05.812440201 +0000 UTC m=+1710.678951420"
Sep 09 16:20:06.395901 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:06.395801    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7698-216eee"]
Sep 09 16:20:06.401552 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:06.401521    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7698-216eee"]
Sep 09 16:20:06.519524 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:06.519490    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7698-9acbd9"]
Sep 09 16:20:06.519853 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:06.519840    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7785ee09-c3da-44fa-b208-88de0d5416ee" containerName="test-curl-7698-216eee"
Sep 09 16:20:06.519898 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:06.519854    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7785ee09-c3da-44fa-b208-88de0d5416ee" containerName="test-curl-7698-216eee"
Sep 09 16:20:06.519950 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:06.519941    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7785ee09-c3da-44fa-b208-88de0d5416ee" containerName="test-curl-7698-216eee"
Sep 09 16:20:06.523051 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:06.523035    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7698-9acbd9"
Sep 09 16:20:06.529713 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:06.529668    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7698-9acbd9"]
Sep 09 16:20:06.611084 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:06.611043    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx4qr\" (UniqueName: \"kubernetes.io/projected/7372fbdd-d8d7-4148-bb12-5b546fca6993-kube-api-access-zx4qr\") pod \"test-curl-7698-9acbd9\" (UID: \"7372fbdd-d8d7-4148-bb12-5b546fca6993\") " pod="opendatahub/test-curl-7698-9acbd9"
Sep 09 16:20:06.712318 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:06.712277    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx4qr\" (UniqueName: \"kubernetes.io/projected/7372fbdd-d8d7-4148-bb12-5b546fca6993-kube-api-access-zx4qr\") pod \"test-curl-7698-9acbd9\" (UID: \"7372fbdd-d8d7-4148-bb12-5b546fca6993\") " pod="opendatahub/test-curl-7698-9acbd9"
Sep 09 16:20:06.720655 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:06.720620    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx4qr\" (UniqueName: \"kubernetes.io/projected/7372fbdd-d8d7-4148-bb12-5b546fca6993-kube-api-access-zx4qr\") pod \"test-curl-7698-9acbd9\" (UID: \"7372fbdd-d8d7-4148-bb12-5b546fca6993\") " pod="opendatahub/test-curl-7698-9acbd9"
Sep 09 16:20:06.804497 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:20:06.804454    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7698-216eee" container="test-curl-7698-216eee"
Sep 09 16:20:06.832750 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:06.832720    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7698-9acbd9"
Sep 09 16:20:07.102084 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:07.102058    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7698-9acbd9"]
Sep 09 16:20:07.104397 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:20:07.104371    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7372fbdd_d8d7_4148_bb12_5b546fca6993.slice/crio-8cf54d38224b303ad45caa4814b8928fd9c7e82794b3af67e4f4bdb3c663a863 WatchSource:0}: Error finding container 8cf54d38224b303ad45caa4814b8928fd9c7e82794b3af67e4f4bdb3c663a863: Status 404 returned error can't find the container with id 8cf54d38224b303ad45caa4814b8928fd9c7e82794b3af67e4f4bdb3c663a863
Sep 09 16:20:07.810532 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:07.810494    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7698-9acbd9" event={"ID":"7372fbdd-d8d7-4148-bb12-5b546fca6993","Type":"ContainerStarted","Data":"dd33a704725821cc9088fc9dfe0c8628b3c2690bfc7ffa1ec3b86a48c4bf267b"}
Sep 09 16:20:07.810987 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:07.810542    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7698-9acbd9" event={"ID":"7372fbdd-d8d7-4148-bb12-5b546fca6993","Type":"ContainerStarted","Data":"8cf54d38224b303ad45caa4814b8928fd9c7e82794b3af67e4f4bdb3c663a863"}
Sep 09 16:20:07.810987 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:07.810674    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7698-216eee" podUID="7785ee09-c3da-44fa-b208-88de0d5416ee" containerName="test-curl-7698-216eee" containerID="cri-o://44ef574207c0a290f6905351b0f17a31306ecdb9c62bc6eb3c5209b7a1d6a9be" gracePeriod=2
Sep 09 16:20:07.824631 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:07.824585    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7698-9acbd9" podStartSLOduration=1.824571654 podStartE2EDuration="1.824571654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:20:07.823302084 +0000 UTC m=+1712.689813296" watchObservedRunningTime="2026-09-09 16:20:07.824571654 +0000 UTC m=+1712.691082874"
Sep 09 16:20:08.418711 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:08.418669    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7698-9acbd9"]
Sep 09 16:20:08.424670 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:08.424645    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7698-9acbd9"]
Sep 09 16:20:08.815606 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:20:08.815573    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7698-9acbd9" container="test-curl-7698-9acbd9"
Sep 09 16:20:09.821271 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:09.821224    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7698-9acbd9" podUID="7372fbdd-d8d7-4148-bb12-5b546fca6993" containerName="test-curl-7698-9acbd9" containerID="cri-o://dd33a704725821cc9088fc9dfe0c8628b3c2690bfc7ffa1ec3b86a48c4bf267b" gracePeriod=2
Sep 09 16:20:09.970945 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:09.970920    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7698-216eee"
Sep 09 16:20:10.040807 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:10.040770    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcnr7\" (UniqueName: \"kubernetes.io/projected/7785ee09-c3da-44fa-b208-88de0d5416ee-kube-api-access-gcnr7\") pod \"7785ee09-c3da-44fa-b208-88de0d5416ee\" (UID: \"7785ee09-c3da-44fa-b208-88de0d5416ee\") "
Sep 09 16:20:10.042834 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:10.042812    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7785ee09-c3da-44fa-b208-88de0d5416ee-kube-api-access-gcnr7" (OuterVolumeSpecName: "kube-api-access-gcnr7") pod "7785ee09-c3da-44fa-b208-88de0d5416ee" (UID: "7785ee09-c3da-44fa-b208-88de0d5416ee"). InnerVolumeSpecName "kube-api-access-gcnr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:20:10.141941 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:10.141857    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcnr7\" (UniqueName: \"kubernetes.io/projected/7785ee09-c3da-44fa-b208-88de0d5416ee-kube-api-access-gcnr7\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:20:10.181527 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:10.181495    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:10.401418 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:10.401325    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:10.826905 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:10.826869    2556 generic.go:356] "Generic (PLEG): container finished" podID="7785ee09-c3da-44fa-b208-88de0d5416ee" containerID="44ef574207c0a290f6905351b0f17a31306ecdb9c62bc6eb3c5209b7a1d6a9be" exitCode=137
Sep 09 16:20:10.827375 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:10.826913    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7698-216eee"
Sep 09 16:20:10.827375 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:10.826980    2556 scope.go:117] "RemoveContainer" containerID="44ef574207c0a290f6905351b0f17a31306ecdb9c62bc6eb3c5209b7a1d6a9be"
Sep 09 16:20:10.836301 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:10.836281    2556 scope.go:117] "RemoveContainer" containerID="44ef574207c0a290f6905351b0f17a31306ecdb9c62bc6eb3c5209b7a1d6a9be"
Sep 09 16:20:10.836665 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:20:10.836646    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ef574207c0a290f6905351b0f17a31306ecdb9c62bc6eb3c5209b7a1d6a9be\": container with ID starting with 44ef574207c0a290f6905351b0f17a31306ecdb9c62bc6eb3c5209b7a1d6a9be not found: ID does not exist" containerID="44ef574207c0a290f6905351b0f17a31306ecdb9c62bc6eb3c5209b7a1d6a9be"
Sep 09 16:20:10.836725 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:10.836675    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ef574207c0a290f6905351b0f17a31306ecdb9c62bc6eb3c5209b7a1d6a9be"} err="failed to get container status \"44ef574207c0a290f6905351b0f17a31306ecdb9c62bc6eb3c5209b7a1d6a9be\": rpc error: code = NotFound desc = could not find container \"44ef574207c0a290f6905351b0f17a31306ecdb9c62bc6eb3c5209b7a1d6a9be\": container with ID starting with 44ef574207c0a290f6905351b0f17a31306ecdb9c62bc6eb3c5209b7a1d6a9be not found: ID does not exist"
Sep 09 16:20:10.836990 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:20:10.836966    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7698-216eee\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="7785ee09-c3da-44fa-b208-88de0d5416ee" pod="opendatahub/test-curl-7698-216eee"
Sep 09 16:20:11.539729 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:11.539671    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:11.763731 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:11.763698    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7785ee09-c3da-44fa-b208-88de0d5416ee" path="/var/lib/kubelet/pods/7785ee09-c3da-44fa-b208-88de0d5416ee/volumes"
Sep 09 16:20:11.998263 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:11.998239    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7698-9acbd9"
Sep 09 16:20:12.057396 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:12.057362    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx4qr\" (UniqueName: \"kubernetes.io/projected/7372fbdd-d8d7-4148-bb12-5b546fca6993-kube-api-access-zx4qr\") pod \"7372fbdd-d8d7-4148-bb12-5b546fca6993\" (UID: \"7372fbdd-d8d7-4148-bb12-5b546fca6993\") "
Sep 09 16:20:12.059396 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:12.059370    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7372fbdd-d8d7-4148-bb12-5b546fca6993-kube-api-access-zx4qr" (OuterVolumeSpecName: "kube-api-access-zx4qr") pod "7372fbdd-d8d7-4148-bb12-5b546fca6993" (UID: "7372fbdd-d8d7-4148-bb12-5b546fca6993"). InnerVolumeSpecName "kube-api-access-zx4qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:20:12.158924 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:12.158837    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx4qr\" (UniqueName: \"kubernetes.io/projected/7372fbdd-d8d7-4148-bb12-5b546fca6993-kube-api-access-zx4qr\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:20:12.822009 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:12.821972    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:12.839179 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:12.839148    2556 generic.go:356] "Generic (PLEG): container finished" podID="7372fbdd-d8d7-4148-bb12-5b546fca6993" containerID="dd33a704725821cc9088fc9dfe0c8628b3c2690bfc7ffa1ec3b86a48c4bf267b" exitCode=137
Sep 09 16:20:12.839355 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:12.839188    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7698-9acbd9"
Sep 09 16:20:12.839355 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:12.839236    2556 scope.go:117] "RemoveContainer" containerID="dd33a704725821cc9088fc9dfe0c8628b3c2690bfc7ffa1ec3b86a48c4bf267b"
Sep 09 16:20:12.847998 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:12.847982    2556 scope.go:117] "RemoveContainer" containerID="dd33a704725821cc9088fc9dfe0c8628b3c2690bfc7ffa1ec3b86a48c4bf267b"
Sep 09 16:20:12.848250 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:20:12.848231    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd33a704725821cc9088fc9dfe0c8628b3c2690bfc7ffa1ec3b86a48c4bf267b\": container with ID starting with dd33a704725821cc9088fc9dfe0c8628b3c2690bfc7ffa1ec3b86a48c4bf267b not found: ID does not exist" containerID="dd33a704725821cc9088fc9dfe0c8628b3c2690bfc7ffa1ec3b86a48c4bf267b"
Sep 09 16:20:12.848320 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:12.848260    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd33a704725821cc9088fc9dfe0c8628b3c2690bfc7ffa1ec3b86a48c4bf267b"} err="failed to get container status \"dd33a704725821cc9088fc9dfe0c8628b3c2690bfc7ffa1ec3b86a48c4bf267b\": rpc error: code = NotFound desc = could not find container \"dd33a704725821cc9088fc9dfe0c8628b3c2690bfc7ffa1ec3b86a48c4bf267b\": container with ID starting with dd33a704725821cc9088fc9dfe0c8628b3c2690bfc7ffa1ec3b86a48c4bf267b not found: ID does not exist"
Sep 09 16:20:12.849020 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:20:12.848998    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7698-9acbd9\" is forbidden: User \"system:node:ip-10-0-133-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-93.ec2.internal' and this object" podUID="7372fbdd-d8d7-4148-bb12-5b546fca6993" pod="opendatahub/test-curl-7698-9acbd9"
Sep 09 16:20:13.764239 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:13.764204    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7372fbdd-d8d7-4148-bb12-5b546fca6993" path="/var/lib/kubelet/pods/7372fbdd-d8d7-4148-bb12-5b546fca6993/volumes"
Sep 09 16:20:15.923866 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:15.923832    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:20.440670 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.440634    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"]
Sep 09 16:20:20.441046 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.441011    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7372fbdd-d8d7-4148-bb12-5b546fca6993" containerName="test-curl-7698-9acbd9"
Sep 09 16:20:20.441046 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.441022    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7372fbdd-d8d7-4148-bb12-5b546fca6993" containerName="test-curl-7698-9acbd9"
Sep 09 16:20:20.441126 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.441078    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7372fbdd-d8d7-4148-bb12-5b546fca6993" containerName="test-curl-7698-9acbd9"
Sep 09 16:20:20.445278 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.445259    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.447561 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.447537    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-131689\"/\"openshift-service-ca.crt\""
Sep 09 16:20:20.448353 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.448324    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-131689\"/\"default-dockercfg-m6knh\""
Sep 09 16:20:20.448486 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.448382    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-131689\"/\"kube-root-ca.crt\""
Sep 09 16:20:20.448552 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.448475    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-131689\"/\"rate-test-model-131689-kserve-self-signed-certs\""
Sep 09 16:20:20.454648 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.454626    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"]
Sep 09 16:20:20.467458 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.467424    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:20.529730 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.529690    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-tmp-dir\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.529730 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.529732    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-home\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.529947 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.529752    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-dshm\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.529947 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.529861    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d915ac-7433-4814-89ef-2edef43f635a-tls-certs\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.529947 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.529905    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-model-cache\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.529947 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.529943    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwvqr\" (UniqueName: \"kubernetes.io/projected/d5d915ac-7433-4814-89ef-2edef43f635a-kube-api-access-qwvqr\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.631263 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.631222    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-model-cache\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.631263 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.631267    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwvqr\" (UniqueName: \"kubernetes.io/projected/d5d915ac-7433-4814-89ef-2edef43f635a-kube-api-access-qwvqr\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.631539 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.631300    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-tmp-dir\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.631539 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.631318    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-home\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.631539 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.631334    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-dshm\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.631539 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.631375    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d915ac-7433-4814-89ef-2edef43f635a-tls-certs\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.631744 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.631697    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-model-cache\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.631791 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.631749    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-tmp-dir\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.631791 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.631775    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-home\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.633627 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.633597    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-dshm\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.633920 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.633899    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d915ac-7433-4814-89ef-2edef43f635a-tls-certs\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.638944 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.638923    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwvqr\" (UniqueName: \"kubernetes.io/projected/d5d915ac-7433-4814-89ef-2edef43f635a-kube-api-access-qwvqr\") pod \"rate-test-model-131689-kserve-6c68788687-ttp7s\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") " pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:20.756659 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:20.756564    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:21.029541 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:21.029515    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"]
Sep 09 16:20:21.032114 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:20:21.032088    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5d915ac_7433_4814_89ef_2edef43f635a.slice/crio-ed75c5928fd037fb0b28edf259ae9aec905131e686d5c265068dff7f66e09bf6 WatchSource:0}: Error finding container ed75c5928fd037fb0b28edf259ae9aec905131e686d5c265068dff7f66e09bf6: Status 404 returned error can't find the container with id ed75c5928fd037fb0b28edf259ae9aec905131e686d5c265068dff7f66e09bf6
Sep 09 16:20:21.883204 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:21.883168    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s" event={"ID":"d5d915ac-7433-4814-89ef-2edef43f635a","Type":"ContainerStarted","Data":"3fb45af8be70cd9c624ebe7452b75cf3a90e30143e30bfb08dd94c8c22836f49"}
Sep 09 16:20:21.883649 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:21.883213    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s" event={"ID":"d5d915ac-7433-4814-89ef-2edef43f635a","Type":"ContainerStarted","Data":"ed75c5928fd037fb0b28edf259ae9aec905131e686d5c265068dff7f66e09bf6"}
Sep 09 16:20:21.903140 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:21.903082    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s" podStartSLOduration=1.903063731 podStartE2EDuration="1.903063731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:20:21.901143483 +0000 UTC m=+1726.767654702" watchObservedRunningTime="2026-09-09 16:20:21.903063731 +0000 UTC m=+1726.769575013"
Sep 09 16:20:27.583580 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:27.583537    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:30.757103 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:30.757064    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:30.757531 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:30.757126    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:30.769538 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:30.769515    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:30.937483 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:30.937445    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:20:31.330803 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.330767    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:31.356609 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.356577    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"]
Sep 09 16:20:31.356869 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.356838    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv" podUID="b08ede8c-1356-42eb-856f-b3264fd620b6" containerName="payload-processing" containerID="cri-o://94b68a31e8a15a0aa0a6ded0ec9ecc1140324d5b546b0cbd8673a299e7642755" gracePeriod=30
Sep 09 16:20:31.365598 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.365571    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"]
Sep 09 16:20:31.365833 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.365812    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc" podUID="99a779c8-7c00-4d90-a1ac-871de28e796c" containerName="payload-pre-processing" containerID="cri-o://6a801dfba907e7bf5f3fed36650f743ac13b8ede74aaa01c995d3fea24976e13" gracePeriod=30
Sep 09 16:20:31.400664 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.400625    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:31.464608 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.464572    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:31.529360 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.529328    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:31.587593 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.587499    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:31.630479 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.630444    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"
Sep 09 16:20:31.633498 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.633477    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"
Sep 09 16:20:31.736268 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.736225    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tknfr\" (UniqueName: \"kubernetes.io/projected/b08ede8c-1356-42eb-856f-b3264fd620b6-kube-api-access-tknfr\") pod \"b08ede8c-1356-42eb-856f-b3264fd620b6\" (UID: \"b08ede8c-1356-42eb-856f-b3264fd620b6\") "
Sep 09 16:20:31.736268 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.736270    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgwl9\" (UniqueName: \"kubernetes.io/projected/99a779c8-7c00-4d90-a1ac-871de28e796c-kube-api-access-lgwl9\") pod \"99a779c8-7c00-4d90-a1ac-871de28e796c\" (UID: \"99a779c8-7c00-4d90-a1ac-871de28e796c\") "
Sep 09 16:20:31.736522 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.736300    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a779c8-7c00-4d90-a1ac-871de28e796c-plugins-config-volume\") pod \"99a779c8-7c00-4d90-a1ac-871de28e796c\" (UID: \"99a779c8-7c00-4d90-a1ac-871de28e796c\") "
Sep 09 16:20:31.736522 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.736393    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08ede8c-1356-42eb-856f-b3264fd620b6-plugins-config-volume\") pod \"b08ede8c-1356-42eb-856f-b3264fd620b6\" (UID: \"b08ede8c-1356-42eb-856f-b3264fd620b6\") "
Sep 09 16:20:31.736717 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.736692    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a779c8-7c00-4d90-a1ac-871de28e796c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99a779c8-7c00-4d90-a1ac-871de28e796c" (UID: "99a779c8-7c00-4d90-a1ac-871de28e796c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:20:31.736822 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.736779    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08ede8c-1356-42eb-856f-b3264fd620b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b08ede8c-1356-42eb-856f-b3264fd620b6" (UID: "b08ede8c-1356-42eb-856f-b3264fd620b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:20:31.738320 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.738298    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a779c8-7c00-4d90-a1ac-871de28e796c-kube-api-access-lgwl9" (OuterVolumeSpecName: "kube-api-access-lgwl9") pod "99a779c8-7c00-4d90-a1ac-871de28e796c" (UID: "99a779c8-7c00-4d90-a1ac-871de28e796c"). InnerVolumeSpecName "kube-api-access-lgwl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:20:31.738401 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.738375    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08ede8c-1356-42eb-856f-b3264fd620b6-kube-api-access-tknfr" (OuterVolumeSpecName: "kube-api-access-tknfr") pod "b08ede8c-1356-42eb-856f-b3264fd620b6" (UID: "b08ede8c-1356-42eb-856f-b3264fd620b6"). InnerVolumeSpecName "kube-api-access-tknfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:20:31.837342 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.837304    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08ede8c-1356-42eb-856f-b3264fd620b6-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:20:31.837342 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.837336    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tknfr\" (UniqueName: \"kubernetes.io/projected/b08ede8c-1356-42eb-856f-b3264fd620b6-kube-api-access-tknfr\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:20:31.837342 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.837348    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgwl9\" (UniqueName: \"kubernetes.io/projected/99a779c8-7c00-4d90-a1ac-871de28e796c-kube-api-access-lgwl9\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:20:31.837848 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.837361    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a779c8-7c00-4d90-a1ac-871de28e796c-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:20:31.923481 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.923411    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:31.932588 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.932557    2556 generic.go:356] "Generic (PLEG): container finished" podID="99a779c8-7c00-4d90-a1ac-871de28e796c" containerID="6a801dfba907e7bf5f3fed36650f743ac13b8ede74aaa01c995d3fea24976e13" exitCode=0
Sep 09 16:20:31.932764 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.932618    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"
Sep 09 16:20:31.932764 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.932628    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc" event={"ID":"99a779c8-7c00-4d90-a1ac-871de28e796c","Type":"ContainerDied","Data":"6a801dfba907e7bf5f3fed36650f743ac13b8ede74aaa01c995d3fea24976e13"}
Sep 09 16:20:31.932764 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.932686    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc" event={"ID":"99a779c8-7c00-4d90-a1ac-871de28e796c","Type":"ContainerDied","Data":"ae0bb62d160c201694a2b47b28a7744e0c8e8f7c6d370f10aaa0daa8754613ec"}
Sep 09 16:20:31.932764 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.932709    2556 scope.go:117] "RemoveContainer" containerID="6a801dfba907e7bf5f3fed36650f743ac13b8ede74aaa01c995d3fea24976e13"
Sep 09 16:20:31.934348 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.934324    2556 generic.go:356] "Generic (PLEG): container finished" podID="b08ede8c-1356-42eb-856f-b3264fd620b6" containerID="94b68a31e8a15a0aa0a6ded0ec9ecc1140324d5b546b0cbd8673a299e7642755" exitCode=0
Sep 09 16:20:31.934491 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.934381    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"
Sep 09 16:20:31.934491 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.934413    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv" event={"ID":"b08ede8c-1356-42eb-856f-b3264fd620b6","Type":"ContainerDied","Data":"94b68a31e8a15a0aa0a6ded0ec9ecc1140324d5b546b0cbd8673a299e7642755"}
Sep 09 16:20:31.934491 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.934479    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv" event={"ID":"b08ede8c-1356-42eb-856f-b3264fd620b6","Type":"ContainerDied","Data":"dccfed70cb3c1b1199830cb8ebbbfbb510b2263160921c938dc35b89e944138e"}
Sep 09 16:20:31.943500 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.943455    2556 scope.go:117] "RemoveContainer" containerID="6a801dfba907e7bf5f3fed36650f743ac13b8ede74aaa01c995d3fea24976e13"
Sep 09 16:20:31.943772 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:20:31.943754    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a801dfba907e7bf5f3fed36650f743ac13b8ede74aaa01c995d3fea24976e13\": container with ID starting with 6a801dfba907e7bf5f3fed36650f743ac13b8ede74aaa01c995d3fea24976e13 not found: ID does not exist" containerID="6a801dfba907e7bf5f3fed36650f743ac13b8ede74aaa01c995d3fea24976e13"
Sep 09 16:20:31.943922 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.943784    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a801dfba907e7bf5f3fed36650f743ac13b8ede74aaa01c995d3fea24976e13"} err="failed to get container status \"6a801dfba907e7bf5f3fed36650f743ac13b8ede74aaa01c995d3fea24976e13\": rpc error: code = NotFound desc = could not find container \"6a801dfba907e7bf5f3fed36650f743ac13b8ede74aaa01c995d3fea24976e13\": container with ID starting with 6a801dfba907e7bf5f3fed36650f743ac13b8ede74aaa01c995d3fea24976e13 not found: ID does not exist"
Sep 09 16:20:31.943922 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.943807    2556 scope.go:117] "RemoveContainer" containerID="94b68a31e8a15a0aa0a6ded0ec9ecc1140324d5b546b0cbd8673a299e7642755"
Sep 09 16:20:31.951864 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.951840    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"]
Sep 09 16:20:31.953345 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.953155    2556 scope.go:117] "RemoveContainer" containerID="94b68a31e8a15a0aa0a6ded0ec9ecc1140324d5b546b0cbd8673a299e7642755"
Sep 09 16:20:31.953514 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:20:31.953486    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b68a31e8a15a0aa0a6ded0ec9ecc1140324d5b546b0cbd8673a299e7642755\": container with ID starting with 94b68a31e8a15a0aa0a6ded0ec9ecc1140324d5b546b0cbd8673a299e7642755 not found: ID does not exist" containerID="94b68a31e8a15a0aa0a6ded0ec9ecc1140324d5b546b0cbd8673a299e7642755"
Sep 09 16:20:31.953629 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.953520    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b68a31e8a15a0aa0a6ded0ec9ecc1140324d5b546b0cbd8673a299e7642755"} err="failed to get container status \"94b68a31e8a15a0aa0a6ded0ec9ecc1140324d5b546b0cbd8673a299e7642755\": rpc error: code = NotFound desc = could not find container \"94b68a31e8a15a0aa0a6ded0ec9ecc1140324d5b546b0cbd8673a299e7642755\": container with ID starting with 94b68a31e8a15a0aa0a6ded0ec9ecc1140324d5b546b0cbd8673a299e7642755 not found: ID does not exist"
Sep 09 16:20:31.955013 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.954993    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94cf14-7d4cb55fbb-5dzvc"]
Sep 09 16:20:31.965681 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.965661    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"]
Sep 09 16:20:31.969584 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:31.969562    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94cf14-5dffc45ff7-q96vv"]
Sep 09 16:20:33.762895 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:33.762863    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99a779c8-7c00-4d90-a1ac-871de28e796c" path="/var/lib/kubelet/pods/99a779c8-7c00-4d90-a1ac-871de28e796c/volumes"
Sep 09 16:20:33.763284 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:33.763208    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b08ede8c-1356-42eb-856f-b3264fd620b6" path="/var/lib/kubelet/pods/b08ede8c-1356-42eb-856f-b3264fd620b6/volumes"
Sep 09 16:20:35.853299 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:35.853267    2556 scope.go:117] "RemoveContainer" containerID="8689fa65dc67b3e4d7f5144367b9b17bd3d9c82387a4eac622fdabd9094710e4"
Sep 09 16:20:37.803567 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:37.803531    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:37.864285 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:37.864249    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:38.082362 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:38.082275    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:38.581769 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:38.581733    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:53.978485 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:53.976367    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:54.002371 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.002335    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"]
Sep 09 16:20:54.002697 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.002670    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6" podUID="ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4" containerName="payload-processing" containerID="cri-o://7efccb1175ccd98102203b98200b82d294d7fd04b8272d43987470bbc7f4d0a2" gracePeriod=30
Sep 09 16:20:54.015062 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.015028    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"]
Sep 09 16:20:54.015323 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.015297    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt" podUID="4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2" containerName="payload-pre-processing" containerID="cri-o://5b4abb14684fd11105e68a803ac2317774d4d475e2d2b8d7c7ebc98ef2da724e" gracePeriod=30
Sep 09 16:20:54.041594 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.041556    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:54.104807 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.104775    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:54.175597 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.175490    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:54.236865 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.236767    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:54.284325 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.284301    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"
Sep 09 16:20:54.287043 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.287026    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"
Sep 09 16:20:54.333870 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.333838    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxd89\" (UniqueName: \"kubernetes.io/projected/ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4-kube-api-access-lxd89\") pod \"ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4\" (UID: \"ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4\") "
Sep 09 16:20:54.334060 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.333903    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2kn7\" (UniqueName: \"kubernetes.io/projected/4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2-kube-api-access-d2kn7\") pod \"4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2\" (UID: \"4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2\") "
Sep 09 16:20:54.334060 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.333936    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2-plugins-config-volume\") pod \"4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2\" (UID: \"4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2\") "
Sep 09 16:20:54.334060 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.333967    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4-plugins-config-volume\") pod \"ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4\" (UID: \"ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4\") "
Sep 09 16:20:54.334308 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.334285    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2" (UID: "4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:20:54.334351 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.334333    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4" (UID: "ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:20:54.335919 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.335899    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2-kube-api-access-d2kn7" (OuterVolumeSpecName: "kube-api-access-d2kn7") pod "4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2" (UID: "4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2"). InnerVolumeSpecName "kube-api-access-d2kn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:20:54.336008 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.335988    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4-kube-api-access-lxd89" (OuterVolumeSpecName: "kube-api-access-lxd89") pod "ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4" (UID: "ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4"). InnerVolumeSpecName "kube-api-access-lxd89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:20:54.434596 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.434554    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:20:54.434596 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.434590    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxd89\" (UniqueName: \"kubernetes.io/projected/ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4-kube-api-access-lxd89\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:20:54.434596 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.434599    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2kn7\" (UniqueName: \"kubernetes.io/projected/4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2-kube-api-access-d2kn7\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:20:54.434596 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.434608    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:20:54.540512 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:54.540409    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:20:55.047583 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.047551    2556 generic.go:356] "Generic (PLEG): container finished" podID="4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2" containerID="5b4abb14684fd11105e68a803ac2317774d4d475e2d2b8d7c7ebc98ef2da724e" exitCode=0
Sep 09 16:20:55.048050 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.047595    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"
Sep 09 16:20:55.048050 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.047640    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt" event={"ID":"4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2","Type":"ContainerDied","Data":"5b4abb14684fd11105e68a803ac2317774d4d475e2d2b8d7c7ebc98ef2da724e"}
Sep 09 16:20:55.048050 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.047705    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt" event={"ID":"4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2","Type":"ContainerDied","Data":"0a52d07aa6cf3b97cc9de497066c3846497a2cf3a91342d9c0bdc3b2a1f87d15"}
Sep 09 16:20:55.048050 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.047740    2556 scope.go:117] "RemoveContainer" containerID="5b4abb14684fd11105e68a803ac2317774d4d475e2d2b8d7c7ebc98ef2da724e"
Sep 09 16:20:55.049242 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.049124    2556 generic.go:356] "Generic (PLEG): container finished" podID="ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4" containerID="7efccb1175ccd98102203b98200b82d294d7fd04b8272d43987470bbc7f4d0a2" exitCode=0
Sep 09 16:20:55.049242 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.049156    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"
Sep 09 16:20:55.049242 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.049185    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6" event={"ID":"ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4","Type":"ContainerDied","Data":"7efccb1175ccd98102203b98200b82d294d7fd04b8272d43987470bbc7f4d0a2"}
Sep 09 16:20:55.049242 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.049226    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6" event={"ID":"ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4","Type":"ContainerDied","Data":"89800f2f5b5e8c933d6e3df706805a3f7c5d03afc2322be1785d3a9a3369d34e"}
Sep 09 16:20:55.056679 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.056652    2556 scope.go:117] "RemoveContainer" containerID="5b4abb14684fd11105e68a803ac2317774d4d475e2d2b8d7c7ebc98ef2da724e"
Sep 09 16:20:55.056958 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:20:55.056936    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b4abb14684fd11105e68a803ac2317774d4d475e2d2b8d7c7ebc98ef2da724e\": container with ID starting with 5b4abb14684fd11105e68a803ac2317774d4d475e2d2b8d7c7ebc98ef2da724e not found: ID does not exist" containerID="5b4abb14684fd11105e68a803ac2317774d4d475e2d2b8d7c7ebc98ef2da724e"
Sep 09 16:20:55.057052 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.056964    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b4abb14684fd11105e68a803ac2317774d4d475e2d2b8d7c7ebc98ef2da724e"} err="failed to get container status \"5b4abb14684fd11105e68a803ac2317774d4d475e2d2b8d7c7ebc98ef2da724e\": rpc error: code = NotFound desc = could not find container \"5b4abb14684fd11105e68a803ac2317774d4d475e2d2b8d7c7ebc98ef2da724e\": container with ID starting with 5b4abb14684fd11105e68a803ac2317774d4d475e2d2b8d7c7ebc98ef2da724e not found: ID does not exist"
Sep 09 16:20:55.057052 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.056984    2556 scope.go:117] "RemoveContainer" containerID="7efccb1175ccd98102203b98200b82d294d7fd04b8272d43987470bbc7f4d0a2"
Sep 09 16:20:55.064894 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.064878    2556 scope.go:117] "RemoveContainer" containerID="7efccb1175ccd98102203b98200b82d294d7fd04b8272d43987470bbc7f4d0a2"
Sep 09 16:20:55.065147 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:20:55.065129    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efccb1175ccd98102203b98200b82d294d7fd04b8272d43987470bbc7f4d0a2\": container with ID starting with 7efccb1175ccd98102203b98200b82d294d7fd04b8272d43987470bbc7f4d0a2 not found: ID does not exist" containerID="7efccb1175ccd98102203b98200b82d294d7fd04b8272d43987470bbc7f4d0a2"
Sep 09 16:20:55.065208 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.065158    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efccb1175ccd98102203b98200b82d294d7fd04b8272d43987470bbc7f4d0a2"} err="failed to get container status \"7efccb1175ccd98102203b98200b82d294d7fd04b8272d43987470bbc7f4d0a2\": rpc error: code = NotFound desc = could not find container \"7efccb1175ccd98102203b98200b82d294d7fd04b8272d43987470bbc7f4d0a2\": container with ID starting with 7efccb1175ccd98102203b98200b82d294d7fd04b8272d43987470bbc7f4d0a2 not found: ID does not exist"
Sep 09 16:20:55.071590 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.071538    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"]
Sep 09 16:20:55.073581 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.073560    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-07cc17-6b845757d-669zt"]
Sep 09 16:20:55.082194 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.082175    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"]
Sep 09 16:20:55.084493 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.084475    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-07cc17-6c9c59b775-wpdq6"]
Sep 09 16:20:55.762768 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.762736    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2" path="/var/lib/kubelet/pods/4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2/volumes"
Sep 09 16:20:55.763082 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:55.763068    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4" path="/var/lib/kubelet/pods/ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4/volumes"
Sep 09 16:20:56.733246 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:20:56.733207    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:00.096610 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:00.096572    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:02.516765 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:02.516718    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:02.569758 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:02.569722    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:11.434008 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.433955    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:11.434407 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.434165    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0ce96-29816175-nknzh"]
Sep 09 16:21:11.438840 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.438815    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0ce96-29816175-nknzh"]
Sep 09 16:21:11.461029 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.460998    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"]
Sep 09 16:21:11.461338 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.461283    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7" podUID="7ec74dd7-e7eb-4405-83b4-fbd754814a39" containerName="payload-processing" containerID="cri-o://773d507526b9d7db58990e0480b5bf4729a91d2c46ee2128b0d8227f8e1525c8" gracePeriod=30
Sep 09 16:21:11.470417 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.470392    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"]
Sep 09 16:21:11.470709 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.470658    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw" podUID="45b26c81-5f3a-489d-95f9-55478cb22fa3" containerName="payload-pre-processing" containerID="cri-o://014bdabfc7c16c0afda84ea9eca3df7f3813f188ece0ecf227bef3be77f903a5" gracePeriod=30
Sep 09 16:21:11.504735 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.504703    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:11.602994 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.602954    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:11.678329 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.678293    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:11.736944 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.736925    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"
Sep 09 16:21:11.739686 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.739665    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"
Sep 09 16:21:11.765352 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.765319    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0169b228-de36-40c9-830b-89fe5a5eaf0c" path="/var/lib/kubelet/pods/0169b228-de36-40c9-830b-89fe5a5eaf0c/volumes"
Sep 09 16:21:11.771184 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.769826    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec74dd7-e7eb-4405-83b4-fbd754814a39-plugins-config-volume\") pod \"7ec74dd7-e7eb-4405-83b4-fbd754814a39\" (UID: \"7ec74dd7-e7eb-4405-83b4-fbd754814a39\") "
Sep 09 16:21:11.771184 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.769868    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzsct\" (UniqueName: \"kubernetes.io/projected/7ec74dd7-e7eb-4405-83b4-fbd754814a39-kube-api-access-tzsct\") pod \"7ec74dd7-e7eb-4405-83b4-fbd754814a39\" (UID: \"7ec74dd7-e7eb-4405-83b4-fbd754814a39\") "
Sep 09 16:21:11.771184 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.769896    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv9fv\" (UniqueName: \"kubernetes.io/projected/45b26c81-5f3a-489d-95f9-55478cb22fa3-kube-api-access-qv9fv\") pod \"45b26c81-5f3a-489d-95f9-55478cb22fa3\" (UID: \"45b26c81-5f3a-489d-95f9-55478cb22fa3\") "
Sep 09 16:21:11.771184 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.769990    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b26c81-5f3a-489d-95f9-55478cb22fa3-plugins-config-volume\") pod \"45b26c81-5f3a-489d-95f9-55478cb22fa3\" (UID: \"45b26c81-5f3a-489d-95f9-55478cb22fa3\") "
Sep 09 16:21:11.771184 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.770576    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b26c81-5f3a-489d-95f9-55478cb22fa3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45b26c81-5f3a-489d-95f9-55478cb22fa3" (UID: "45b26c81-5f3a-489d-95f9-55478cb22fa3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:21:11.771184 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.770975    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec74dd7-e7eb-4405-83b4-fbd754814a39-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ec74dd7-e7eb-4405-83b4-fbd754814a39" (UID: "7ec74dd7-e7eb-4405-83b4-fbd754814a39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:21:11.773343 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.773317    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec74dd7-e7eb-4405-83b4-fbd754814a39-kube-api-access-tzsct" (OuterVolumeSpecName: "kube-api-access-tzsct") pod "7ec74dd7-e7eb-4405-83b4-fbd754814a39" (UID: "7ec74dd7-e7eb-4405-83b4-fbd754814a39"). InnerVolumeSpecName "kube-api-access-tzsct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:21:11.774063 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.774029    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b26c81-5f3a-489d-95f9-55478cb22fa3-kube-api-access-qv9fv" (OuterVolumeSpecName: "kube-api-access-qv9fv") pod "45b26c81-5f3a-489d-95f9-55478cb22fa3" (UID: "45b26c81-5f3a-489d-95f9-55478cb22fa3"). InnerVolumeSpecName "kube-api-access-qv9fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:21:11.871450 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.871416    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b26c81-5f3a-489d-95f9-55478cb22fa3-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:21:11.871450 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.871443    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec74dd7-e7eb-4405-83b4-fbd754814a39-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:21:11.871450 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.871453    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzsct\" (UniqueName: \"kubernetes.io/projected/7ec74dd7-e7eb-4405-83b4-fbd754814a39-kube-api-access-tzsct\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:21:11.871705 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:11.871482    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv9fv\" (UniqueName: \"kubernetes.io/projected/45b26c81-5f3a-489d-95f9-55478cb22fa3-kube-api-access-qv9fv\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:21:12.015805 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.015712    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:12.127257 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.127219    2556 generic.go:356] "Generic (PLEG): container finished" podID="45b26c81-5f3a-489d-95f9-55478cb22fa3" containerID="014bdabfc7c16c0afda84ea9eca3df7f3813f188ece0ecf227bef3be77f903a5" exitCode=0
Sep 09 16:21:12.127426 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.127263    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"
Sep 09 16:21:12.127426 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.127309    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw" event={"ID":"45b26c81-5f3a-489d-95f9-55478cb22fa3","Type":"ContainerDied","Data":"014bdabfc7c16c0afda84ea9eca3df7f3813f188ece0ecf227bef3be77f903a5"}
Sep 09 16:21:12.127426 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.127364    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw" event={"ID":"45b26c81-5f3a-489d-95f9-55478cb22fa3","Type":"ContainerDied","Data":"ddda6e8850c1bc3dd805f07024764b7dad63902d0ea3dd19d43fc32f9a39f84a"}
Sep 09 16:21:12.127426 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.127396    2556 scope.go:117] "RemoveContainer" containerID="014bdabfc7c16c0afda84ea9eca3df7f3813f188ece0ecf227bef3be77f903a5"
Sep 09 16:21:12.128705 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.128683    2556 generic.go:356] "Generic (PLEG): container finished" podID="7ec74dd7-e7eb-4405-83b4-fbd754814a39" containerID="773d507526b9d7db58990e0480b5bf4729a91d2c46ee2128b0d8227f8e1525c8" exitCode=0
Sep 09 16:21:12.128809 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.128721    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7" event={"ID":"7ec74dd7-e7eb-4405-83b4-fbd754814a39","Type":"ContainerDied","Data":"773d507526b9d7db58990e0480b5bf4729a91d2c46ee2128b0d8227f8e1525c8"}
Sep 09 16:21:12.128809 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.128766    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7" event={"ID":"7ec74dd7-e7eb-4405-83b4-fbd754814a39","Type":"ContainerDied","Data":"cf011deea190b45718adb6298ab5a52f3829d4a6eaee8305049b1aa85e5533af"}
Sep 09 16:21:12.128809 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.128729    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"
Sep 09 16:21:12.136570 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.136389    2556 scope.go:117] "RemoveContainer" containerID="014bdabfc7c16c0afda84ea9eca3df7f3813f188ece0ecf227bef3be77f903a5"
Sep 09 16:21:12.136846 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:21:12.136816    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014bdabfc7c16c0afda84ea9eca3df7f3813f188ece0ecf227bef3be77f903a5\": container with ID starting with 014bdabfc7c16c0afda84ea9eca3df7f3813f188ece0ecf227bef3be77f903a5 not found: ID does not exist" containerID="014bdabfc7c16c0afda84ea9eca3df7f3813f188ece0ecf227bef3be77f903a5"
Sep 09 16:21:12.136933 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.136859    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014bdabfc7c16c0afda84ea9eca3df7f3813f188ece0ecf227bef3be77f903a5"} err="failed to get container status \"014bdabfc7c16c0afda84ea9eca3df7f3813f188ece0ecf227bef3be77f903a5\": rpc error: code = NotFound desc = could not find container \"014bdabfc7c16c0afda84ea9eca3df7f3813f188ece0ecf227bef3be77f903a5\": container with ID starting with 014bdabfc7c16c0afda84ea9eca3df7f3813f188ece0ecf227bef3be77f903a5 not found: ID does not exist"
Sep 09 16:21:12.136933 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.136879    2556 scope.go:117] "RemoveContainer" containerID="773d507526b9d7db58990e0480b5bf4729a91d2c46ee2128b0d8227f8e1525c8"
Sep 09 16:21:12.144522 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.144505    2556 scope.go:117] "RemoveContainer" containerID="773d507526b9d7db58990e0480b5bf4729a91d2c46ee2128b0d8227f8e1525c8"
Sep 09 16:21:12.144769 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:21:12.144750    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773d507526b9d7db58990e0480b5bf4729a91d2c46ee2128b0d8227f8e1525c8\": container with ID starting with 773d507526b9d7db58990e0480b5bf4729a91d2c46ee2128b0d8227f8e1525c8 not found: ID does not exist" containerID="773d507526b9d7db58990e0480b5bf4729a91d2c46ee2128b0d8227f8e1525c8"
Sep 09 16:21:12.144821 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.144775    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773d507526b9d7db58990e0480b5bf4729a91d2c46ee2128b0d8227f8e1525c8"} err="failed to get container status \"773d507526b9d7db58990e0480b5bf4729a91d2c46ee2128b0d8227f8e1525c8\": rpc error: code = NotFound desc = could not find container \"773d507526b9d7db58990e0480b5bf4729a91d2c46ee2128b0d8227f8e1525c8\": container with ID starting with 773d507526b9d7db58990e0480b5bf4729a91d2c46ee2128b0d8227f8e1525c8 not found: ID does not exist"
Sep 09 16:21:12.150639 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.150617    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"]
Sep 09 16:21:12.154266 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.154240    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0ce96-7998f89d4b-v2jcw"]
Sep 09 16:21:12.163882 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.163859    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"]
Sep 09 16:21:12.167350 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:12.167328    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0ce96-684b9c9f79-ln4k7"]
Sep 09 16:21:13.763333 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:13.763300    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45b26c81-5f3a-489d-95f9-55478cb22fa3" path="/var/lib/kubelet/pods/45b26c81-5f3a-489d-95f9-55478cb22fa3/volumes"
Sep 09 16:21:13.763732 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:13.763666    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ec74dd7-e7eb-4405-83b4-fbd754814a39" path="/var/lib/kubelet/pods/7ec74dd7-e7eb-4405-83b4-fbd754814a39/volumes"
Sep 09 16:21:14.350248 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:14.350209    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:17.670190 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:17.670152    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:17.771018 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:17.770981    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:17.937030 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:17.936993    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:18.974399 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:18.974360    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:28.886020 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:28.885934    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:28.920132 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:28.920096    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"]
Sep 09 16:21:28.986388 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:28.986349    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:29.074093 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:29.074038    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:29.125872 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:29.125839    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:29.212004 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:29.211971    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt" podUID="d11392f7-b713-4d01-8a3c-c9e865560acd" containerName="payload-pre-processing" containerID="cri-o://440b333b06dbe901e0252d5bb5310677c78f2f1d6c28aea157501fc1d6ca0630" gracePeriod=30
Sep 09 16:21:29.344808 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:29.344768    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt" podUID="d11392f7-b713-4d01-8a3c-c9e865560acd" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.60:9004: connect: connection refused"
Sep 09 16:21:29.441130 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:29.441097    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:29.456952 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:29.456924    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"
Sep 09 16:21:29.515220 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:29.515143    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11392f7-b713-4d01-8a3c-c9e865560acd-plugins-config-volume\") pod \"d11392f7-b713-4d01-8a3c-c9e865560acd\" (UID: \"d11392f7-b713-4d01-8a3c-c9e865560acd\") "
Sep 09 16:21:29.515356 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:29.515290    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gl66s\" (UniqueName: \"kubernetes.io/projected/d11392f7-b713-4d01-8a3c-c9e865560acd-kube-api-access-gl66s\") pod \"d11392f7-b713-4d01-8a3c-c9e865560acd\" (UID: \"d11392f7-b713-4d01-8a3c-c9e865560acd\") "
Sep 09 16:21:29.515612 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:29.515591    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d11392f7-b713-4d01-8a3c-c9e865560acd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d11392f7-b713-4d01-8a3c-c9e865560acd" (UID: "d11392f7-b713-4d01-8a3c-c9e865560acd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:21:29.517266 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:29.517245    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11392f7-b713-4d01-8a3c-c9e865560acd-kube-api-access-gl66s" (OuterVolumeSpecName: "kube-api-access-gl66s") pod "d11392f7-b713-4d01-8a3c-c9e865560acd" (UID: "d11392f7-b713-4d01-8a3c-c9e865560acd"). InnerVolumeSpecName "kube-api-access-gl66s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:21:29.616795 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:29.616763    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl66s\" (UniqueName: \"kubernetes.io/projected/d11392f7-b713-4d01-8a3c-c9e865560acd-kube-api-access-gl66s\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:21:29.616795 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:29.616790    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11392f7-b713-4d01-8a3c-c9e865560acd-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:21:30.218339 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:30.218287    2556 generic.go:356] "Generic (PLEG): container finished" podID="d11392f7-b713-4d01-8a3c-c9e865560acd" containerID="440b333b06dbe901e0252d5bb5310677c78f2f1d6c28aea157501fc1d6ca0630" exitCode=0
Sep 09 16:21:30.218339 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:30.218335    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"
Sep 09 16:21:30.218888 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:30.218358    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt" event={"ID":"d11392f7-b713-4d01-8a3c-c9e865560acd","Type":"ContainerDied","Data":"440b333b06dbe901e0252d5bb5310677c78f2f1d6c28aea157501fc1d6ca0630"}
Sep 09 16:21:30.218888 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:30.218410    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt" event={"ID":"d11392f7-b713-4d01-8a3c-c9e865560acd","Type":"ContainerDied","Data":"ec891e025e4b24df381e2c84d989d797165139be39500e45cf3a599c065fbc48"}
Sep 09 16:21:30.218888 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:30.218431    2556 scope.go:117] "RemoveContainer" containerID="440b333b06dbe901e0252d5bb5310677c78f2f1d6c28aea157501fc1d6ca0630"
Sep 09 16:21:30.227115 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:30.227095    2556 scope.go:117] "RemoveContainer" containerID="440b333b06dbe901e0252d5bb5310677c78f2f1d6c28aea157501fc1d6ca0630"
Sep 09 16:21:30.227398 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:21:30.227379    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440b333b06dbe901e0252d5bb5310677c78f2f1d6c28aea157501fc1d6ca0630\": container with ID starting with 440b333b06dbe901e0252d5bb5310677c78f2f1d6c28aea157501fc1d6ca0630 not found: ID does not exist" containerID="440b333b06dbe901e0252d5bb5310677c78f2f1d6c28aea157501fc1d6ca0630"
Sep 09 16:21:30.227477 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:30.227406    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440b333b06dbe901e0252d5bb5310677c78f2f1d6c28aea157501fc1d6ca0630"} err="failed to get container status \"440b333b06dbe901e0252d5bb5310677c78f2f1d6c28aea157501fc1d6ca0630\": rpc error: code = NotFound desc = could not find container \"440b333b06dbe901e0252d5bb5310677c78f2f1d6c28aea157501fc1d6ca0630\": container with ID starting with 440b333b06dbe901e0252d5bb5310677c78f2f1d6c28aea157501fc1d6ca0630 not found: ID does not exist"
Sep 09 16:21:30.233995 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:30.233969    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"]
Sep 09 16:21:30.237646 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:30.237624    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-008161-84f7cd57d-2tqwt"]
Sep 09 16:21:31.763702 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:31.763669    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d11392f7-b713-4d01-8a3c-c9e865560acd" path="/var/lib/kubelet/pods/d11392f7-b713-4d01-8a3c-c9e865560acd/volumes"
Sep 09 16:21:35.475317 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:35.475279    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:35.569200 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:35.569156    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:35.718572 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:35.718545    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 16:21:35.721239 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:35.721216    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 16:21:35.737639 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:35.737574    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:35.885573 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:35.885539    2556 scope.go:117] "RemoveContainer" containerID="ba3a5ff04835a86f0980f0d99fec5b5a1845f5853ed156c5cebe236555b95356"
Sep 09 16:21:51.488607 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:51.488566    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"]
Sep 09 16:21:51.513740 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:51.513706    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:52.463152 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:52.463117    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:52.924234 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:52.924195    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:53.252531 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:53.252431    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:53.434286 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:53.434252    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:56.651964 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:56.651864    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:56.801097 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:56.801060    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:57.886771 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:57.886730    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:58.011721 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.011689    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:58.365840 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.365805    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:58.466214 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466178    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"]
Sep 09 16:21:58.466535 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466520    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec74dd7-e7eb-4405-83b4-fbd754814a39" containerName="payload-processing"
Sep 09 16:21:58.466535 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466535    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec74dd7-e7eb-4405-83b4-fbd754814a39" containerName="payload-processing"
Sep 09 16:21:58.466650 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466560    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99a779c8-7c00-4d90-a1ac-871de28e796c" containerName="payload-pre-processing"
Sep 09 16:21:58.466650 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466569    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="99a779c8-7c00-4d90-a1ac-871de28e796c" containerName="payload-pre-processing"
Sep 09 16:21:58.466650 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466582    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2" containerName="payload-pre-processing"
Sep 09 16:21:58.466650 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466591    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2" containerName="payload-pre-processing"
Sep 09 16:21:58.466650 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466602    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b26c81-5f3a-489d-95f9-55478cb22fa3" containerName="payload-pre-processing"
Sep 09 16:21:58.466650 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466607    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b26c81-5f3a-489d-95f9-55478cb22fa3" containerName="payload-pre-processing"
Sep 09 16:21:58.466650 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466617    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08ede8c-1356-42eb-856f-b3264fd620b6" containerName="payload-processing"
Sep 09 16:21:58.466650 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466622    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08ede8c-1356-42eb-856f-b3264fd620b6" containerName="payload-processing"
Sep 09 16:21:58.466650 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466630    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d11392f7-b713-4d01-8a3c-c9e865560acd" containerName="payload-pre-processing"
Sep 09 16:21:58.466650 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466636    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d11392f7-b713-4d01-8a3c-c9e865560acd" containerName="payload-pre-processing"
Sep 09 16:21:58.466650 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466643    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4" containerName="payload-processing"
Sep 09 16:21:58.466650 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466649    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4" containerName="payload-processing"
Sep 09 16:21:58.467127 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466704    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b26c81-5f3a-489d-95f9-55478cb22fa3" containerName="payload-pre-processing"
Sep 09 16:21:58.467127 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466710    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c1ef883-0ff8-4ca7-9ae5-8a02c03c2ca2" containerName="payload-pre-processing"
Sep 09 16:21:58.467127 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466719    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b08ede8c-1356-42eb-856f-b3264fd620b6" containerName="payload-processing"
Sep 09 16:21:58.467127 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466731    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ec74dd7-e7eb-4405-83b4-fbd754814a39" containerName="payload-processing"
Sep 09 16:21:58.467127 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466740    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d11392f7-b713-4d01-8a3c-c9e865560acd" containerName="payload-pre-processing"
Sep 09 16:21:58.467127 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466745    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="99a779c8-7c00-4d90-a1ac-871de28e796c" containerName="payload-pre-processing"
Sep 09 16:21:58.467127 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466749    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8aca0a-f93a-43cb-8a13-f0bf15b99ef4" containerName="payload-processing"
Sep 09 16:21:58.467127 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.466950    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:58.469655 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.469638    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"
Sep 09 16:21:58.471803 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.471777    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8cc2d6-dockercfg-h54cw\""
Sep 09 16:21:58.471920 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.471850    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8cc2d6\""
Sep 09 16:21:58.479497 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.479459    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"]
Sep 09 16:21:58.541231 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.541188    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:21:58.558495 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.558444    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7efd0e-8b7f-4a46-8203-a268ec09d940-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk\" (UID: \"ac7efd0e-8b7f-4a46-8203-a268ec09d940\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"
Sep 09 16:21:58.558639 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.558503    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lk4z\" (UniqueName: \"kubernetes.io/projected/ac7efd0e-8b7f-4a46-8203-a268ec09d940-kube-api-access-5lk4z\") pod \"payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk\" (UID: \"ac7efd0e-8b7f-4a46-8203-a268ec09d940\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"
Sep 09 16:21:58.659205 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.659108    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7efd0e-8b7f-4a46-8203-a268ec09d940-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk\" (UID: \"ac7efd0e-8b7f-4a46-8203-a268ec09d940\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"
Sep 09 16:21:58.659205 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.659151    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lk4z\" (UniqueName: \"kubernetes.io/projected/ac7efd0e-8b7f-4a46-8203-a268ec09d940-kube-api-access-5lk4z\") pod \"payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk\" (UID: \"ac7efd0e-8b7f-4a46-8203-a268ec09d940\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"
Sep 09 16:21:58.659931 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.659909    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7efd0e-8b7f-4a46-8203-a268ec09d940-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk\" (UID: \"ac7efd0e-8b7f-4a46-8203-a268ec09d940\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"
Sep 09 16:21:58.667625 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.667594    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lk4z\" (UniqueName: \"kubernetes.io/projected/ac7efd0e-8b7f-4a46-8203-a268ec09d940-kube-api-access-5lk4z\") pod \"payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk\" (UID: \"ac7efd0e-8b7f-4a46-8203-a268ec09d940\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"
Sep 09 16:21:58.780934 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:58.780898    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"
Sep 09 16:21:59.051846 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:59.051811    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"]
Sep 09 16:21:59.054948 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:21:59.054918    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7efd0e_8b7f_4a46_8203_a268ec09d940.slice/crio-f0622b5a1e2cb8ebd0c4923b1db8a11879152c5a72ea9bf16f8767792252377a WatchSource:0}: Error finding container f0622b5a1e2cb8ebd0c4923b1db8a11879152c5a72ea9bf16f8767792252377a: Status 404 returned error can't find the container with id f0622b5a1e2cb8ebd0c4923b1db8a11879152c5a72ea9bf16f8767792252377a
Sep 09 16:21:59.352962 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:59.352877    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk" event={"ID":"ac7efd0e-8b7f-4a46-8203-a268ec09d940","Type":"ContainerStarted","Data":"552b55088f2ba33ec4ae7b952e8b37b6bdb8249548f4066128bfd406a3d26221"}
Sep 09 16:21:59.352962 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:59.352924    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk" event={"ID":"ac7efd0e-8b7f-4a46-8203-a268ec09d940","Type":"ContainerStarted","Data":"f0622b5a1e2cb8ebd0c4923b1db8a11879152c5a72ea9bf16f8767792252377a"}
Sep 09 16:21:59.353158 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:59.353057    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"
Sep 09 16:21:59.367882 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:21:59.367832    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk" podStartSLOduration=1.36781607 podStartE2EDuration="1.36781607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:21:59.366777914 +0000 UTC m=+1824.233289149" watchObservedRunningTime="2026-09-09 16:21:59.36781607 +0000 UTC m=+1824.234327290"
Sep 09 16:22:06.508355 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.508306    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s" podUID="d5d915ac-7433-4814-89ef-2edef43f635a" containerName="main" containerID="cri-o://3fb45af8be70cd9c624ebe7452b75cf3a90e30143e30bfb08dd94c8c22836f49" gracePeriod=45
Sep 09 16:22:06.763145 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.763084    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:22:06.831517 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.831488    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwvqr\" (UniqueName: \"kubernetes.io/projected/d5d915ac-7433-4814-89ef-2edef43f635a-kube-api-access-qwvqr\") pod \"d5d915ac-7433-4814-89ef-2edef43f635a\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") "
Sep 09 16:22:06.831701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.831527    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-home\") pod \"d5d915ac-7433-4814-89ef-2edef43f635a\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") "
Sep 09 16:22:06.831701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.831559    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-tmp-dir\") pod \"d5d915ac-7433-4814-89ef-2edef43f635a\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") "
Sep 09 16:22:06.831701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.831589    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-dshm\") pod \"d5d915ac-7433-4814-89ef-2edef43f635a\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") "
Sep 09 16:22:06.831701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.831611    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-model-cache\") pod \"d5d915ac-7433-4814-89ef-2edef43f635a\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") "
Sep 09 16:22:06.831701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.831665    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d915ac-7433-4814-89ef-2edef43f635a-tls-certs\") pod \"d5d915ac-7433-4814-89ef-2edef43f635a\" (UID: \"d5d915ac-7433-4814-89ef-2edef43f635a\") "
Sep 09 16:22:06.831970 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.831827    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5d915ac-7433-4814-89ef-2edef43f635a" (UID: "d5d915ac-7433-4814-89ef-2edef43f635a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:22:06.832029 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.831973    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-tmp-dir\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:22:06.832029 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.832000    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5d915ac-7433-4814-89ef-2edef43f635a" (UID: "d5d915ac-7433-4814-89ef-2edef43f635a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:22:06.832288 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.832256    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-home" (OuterVolumeSpecName: "home") pod "d5d915ac-7433-4814-89ef-2edef43f635a" (UID: "d5d915ac-7433-4814-89ef-2edef43f635a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:22:06.833853 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.833829    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d915ac-7433-4814-89ef-2edef43f635a-kube-api-access-qwvqr" (OuterVolumeSpecName: "kube-api-access-qwvqr") pod "d5d915ac-7433-4814-89ef-2edef43f635a" (UID: "d5d915ac-7433-4814-89ef-2edef43f635a"). InnerVolumeSpecName "kube-api-access-qwvqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:22:06.833953 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.833878    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-dshm" (OuterVolumeSpecName: "dshm") pod "d5d915ac-7433-4814-89ef-2edef43f635a" (UID: "d5d915ac-7433-4814-89ef-2edef43f635a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:22:06.833953 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.833878    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5d915ac-7433-4814-89ef-2edef43f635a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5d915ac-7433-4814-89ef-2edef43f635a" (UID: "d5d915ac-7433-4814-89ef-2edef43f635a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:22:06.932880 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.932850    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwvqr\" (UniqueName: \"kubernetes.io/projected/d5d915ac-7433-4814-89ef-2edef43f635a-kube-api-access-qwvqr\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:22:06.932880 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.932877    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-home\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:22:06.932880 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.932889    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-dshm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:22:06.933108 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.932899    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d915ac-7433-4814-89ef-2edef43f635a-model-cache\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:22:06.933108 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:06.932913    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d915ac-7433-4814-89ef-2edef43f635a-tls-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:22:07.389095 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:07.389063    2556 generic.go:356] "Generic (PLEG): container finished" podID="d5d915ac-7433-4814-89ef-2edef43f635a" containerID="3fb45af8be70cd9c624ebe7452b75cf3a90e30143e30bfb08dd94c8c22836f49" exitCode=0
Sep 09 16:22:07.389279 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:07.389108    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"
Sep 09 16:22:07.389279 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:07.389108    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s" event={"ID":"d5d915ac-7433-4814-89ef-2edef43f635a","Type":"ContainerDied","Data":"3fb45af8be70cd9c624ebe7452b75cf3a90e30143e30bfb08dd94c8c22836f49"}
Sep 09 16:22:07.389279 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:07.389233    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s" event={"ID":"d5d915ac-7433-4814-89ef-2edef43f635a","Type":"ContainerDied","Data":"ed75c5928fd037fb0b28edf259ae9aec905131e686d5c265068dff7f66e09bf6"}
Sep 09 16:22:07.389279 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:07.389254    2556 scope.go:117] "RemoveContainer" containerID="3fb45af8be70cd9c624ebe7452b75cf3a90e30143e30bfb08dd94c8c22836f49"
Sep 09 16:22:07.397882 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:07.397863    2556 scope.go:117] "RemoveContainer" containerID="3fb45af8be70cd9c624ebe7452b75cf3a90e30143e30bfb08dd94c8c22836f49"
Sep 09 16:22:07.398152 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:22:07.398135    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb45af8be70cd9c624ebe7452b75cf3a90e30143e30bfb08dd94c8c22836f49\": container with ID starting with 3fb45af8be70cd9c624ebe7452b75cf3a90e30143e30bfb08dd94c8c22836f49 not found: ID does not exist" containerID="3fb45af8be70cd9c624ebe7452b75cf3a90e30143e30bfb08dd94c8c22836f49"
Sep 09 16:22:07.398198 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:07.398161    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb45af8be70cd9c624ebe7452b75cf3a90e30143e30bfb08dd94c8c22836f49"} err="failed to get container status \"3fb45af8be70cd9c624ebe7452b75cf3a90e30143e30bfb08dd94c8c22836f49\": rpc error: code = NotFound desc = could not find container \"3fb45af8be70cd9c624ebe7452b75cf3a90e30143e30bfb08dd94c8c22836f49\": container with ID starting with 3fb45af8be70cd9c624ebe7452b75cf3a90e30143e30bfb08dd94c8c22836f49 not found: ID does not exist"
Sep 09 16:22:07.408494 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:07.408454    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"]
Sep 09 16:22:07.412596 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:07.412568    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/rate-test-model-131689-kserve-6c68788687-ttp7s"]
Sep 09 16:22:07.763241 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:07.763210    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5d915ac-7433-4814-89ef-2edef43f635a" path="/var/lib/kubelet/pods/d5d915ac-7433-4814-89ef-2edef43f635a/volumes"
Sep 09 16:22:10.358838 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:10.358806    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"
Sep 09 16:22:21.616046 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:21.616014    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:22:25.902786 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:25.902743    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:22:27.160483 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.160428    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:22:27.264968 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.264929    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:22:27.361412 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.361380    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:22:27.628908 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.628872    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:22:27.720370 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.720327    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:22:27.736410 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.736375    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"]
Sep 09 16:22:27.736743 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.736730    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d915ac-7433-4814-89ef-2edef43f635a" containerName="main"
Sep 09 16:22:27.736789 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.736745    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d915ac-7433-4814-89ef-2edef43f635a" containerName="main"
Sep 09 16:22:27.736829 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.736809    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d915ac-7433-4814-89ef-2edef43f635a" containerName="main"
Sep 09 16:22:27.739802 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.739787    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"
Sep 09 16:22:27.741901 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.741884    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b99260-dockercfg-t4497\""
Sep 09 16:22:27.741996 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.741883    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b99260\""
Sep 09 16:22:27.748552 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.748527    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"]
Sep 09 16:22:27.763840 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.763815    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"]
Sep 09 16:22:27.767251 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.767236    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"
Sep 09 16:22:27.773219 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.773197    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"]
Sep 09 16:22:27.818708 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.818675    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ndkz\" (UniqueName: \"kubernetes.io/projected/d3662e0e-d5c5-4f54-96fa-e1da62ba781a-kube-api-access-9ndkz\") pod \"payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd\" (UID: \"d3662e0e-d5c5-4f54-96fa-e1da62ba781a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"
Sep 09 16:22:27.818869 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.818731    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3662e0e-d5c5-4f54-96fa-e1da62ba781a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd\" (UID: \"d3662e0e-d5c5-4f54-96fa-e1da62ba781a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"
Sep 09 16:22:27.920089 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.920004    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3662e0e-d5c5-4f54-96fa-e1da62ba781a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd\" (UID: \"d3662e0e-d5c5-4f54-96fa-e1da62ba781a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"
Sep 09 16:22:27.920269 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.920234    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2fqw\" (UniqueName: \"kubernetes.io/projected/3fcd01e5-8368-471b-9586-5454464f5c81-kube-api-access-d2fqw\") pod \"payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m\" (UID: \"3fcd01e5-8368-471b-9586-5454464f5c81\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"
Sep 09 16:22:27.920413 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.920388    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ndkz\" (UniqueName: \"kubernetes.io/projected/d3662e0e-d5c5-4f54-96fa-e1da62ba781a-kube-api-access-9ndkz\") pod \"payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd\" (UID: \"d3662e0e-d5c5-4f54-96fa-e1da62ba781a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"
Sep 09 16:22:27.920522 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.920432    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fcd01e5-8368-471b-9586-5454464f5c81-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m\" (UID: \"3fcd01e5-8368-471b-9586-5454464f5c81\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"
Sep 09 16:22:27.920939 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.920920    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3662e0e-d5c5-4f54-96fa-e1da62ba781a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd\" (UID: \"d3662e0e-d5c5-4f54-96fa-e1da62ba781a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"
Sep 09 16:22:27.928303 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:27.928274    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ndkz\" (UniqueName: \"kubernetes.io/projected/d3662e0e-d5c5-4f54-96fa-e1da62ba781a-kube-api-access-9ndkz\") pod \"payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd\" (UID: \"d3662e0e-d5c5-4f54-96fa-e1da62ba781a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"
Sep 09 16:22:28.021688 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.021654    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fcd01e5-8368-471b-9586-5454464f5c81-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m\" (UID: \"3fcd01e5-8368-471b-9586-5454464f5c81\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"
Sep 09 16:22:28.021884 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.021730    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2fqw\" (UniqueName: \"kubernetes.io/projected/3fcd01e5-8368-471b-9586-5454464f5c81-kube-api-access-d2fqw\") pod \"payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m\" (UID: \"3fcd01e5-8368-471b-9586-5454464f5c81\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"
Sep 09 16:22:28.022420 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.022398    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fcd01e5-8368-471b-9586-5454464f5c81-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m\" (UID: \"3fcd01e5-8368-471b-9586-5454464f5c81\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"
Sep 09 16:22:28.029109 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.029080    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2fqw\" (UniqueName: \"kubernetes.io/projected/3fcd01e5-8368-471b-9586-5454464f5c81-kube-api-access-d2fqw\") pod \"payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m\" (UID: \"3fcd01e5-8368-471b-9586-5454464f5c81\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"
Sep 09 16:22:28.051639 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.051612    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"
Sep 09 16:22:28.079021 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.078994    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"
Sep 09 16:22:28.323062 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.323027    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"]
Sep 09 16:22:28.326545 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:22:28.326519    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3662e0e_d5c5_4f54_96fa_e1da62ba781a.slice/crio-815899b3455710d8dad437f1f2e8016c266a68d2f1b869573bb05bb4ab5b052b WatchSource:0}: Error finding container 815899b3455710d8dad437f1f2e8016c266a68d2f1b869573bb05bb4ab5b052b: Status 404 returned error can't find the container with id 815899b3455710d8dad437f1f2e8016c266a68d2f1b869573bb05bb4ab5b052b
Sep 09 16:22:28.355415 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.355393    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"]
Sep 09 16:22:28.360607 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:22:28.360582    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fcd01e5_8368_471b_9586_5454464f5c81.slice/crio-a0139042f76476feb7b7f3072e8fe76fc4245d5f9eb77e5cc49be6bf05d84b4f WatchSource:0}: Error finding container a0139042f76476feb7b7f3072e8fe76fc4245d5f9eb77e5cc49be6bf05d84b4f: Status 404 returned error can't find the container with id a0139042f76476feb7b7f3072e8fe76fc4245d5f9eb77e5cc49be6bf05d84b4f
Sep 09 16:22:28.487368 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.487328    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m" event={"ID":"3fcd01e5-8368-471b-9586-5454464f5c81","Type":"ContainerStarted","Data":"bed53938fa8cac1e74940157b411f66b054338cc7fcb92aa0de2209aae3c054a"}
Sep 09 16:22:28.487586 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.487391    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m" event={"ID":"3fcd01e5-8368-471b-9586-5454464f5c81","Type":"ContainerStarted","Data":"a0139042f76476feb7b7f3072e8fe76fc4245d5f9eb77e5cc49be6bf05d84b4f"}
Sep 09 16:22:28.487586 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.487503    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"
Sep 09 16:22:28.489039 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.489014    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd" event={"ID":"d3662e0e-d5c5-4f54-96fa-e1da62ba781a","Type":"ContainerStarted","Data":"6e6651e31abb91f159cba7ec5e16225e804e106fc76a9b3393f368b37b6f2c72"}
Sep 09 16:22:28.489155 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.489050    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd" event={"ID":"d3662e0e-d5c5-4f54-96fa-e1da62ba781a","Type":"ContainerStarted","Data":"815899b3455710d8dad437f1f2e8016c266a68d2f1b869573bb05bb4ab5b052b"}
Sep 09 16:22:28.489155 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.489070    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"
Sep 09 16:22:28.504119 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.504068    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m" podStartSLOduration=1.504054988 podStartE2EDuration="1.504054988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:22:28.501899784 +0000 UTC m=+1853.368411006" watchObservedRunningTime="2026-09-09 16:22:28.504054988 +0000 UTC m=+1853.370566259"
Sep 09 16:22:28.517821 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:28.517770    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd" podStartSLOduration=1.517755489 podStartE2EDuration="1.517755489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:22:28.5167843 +0000 UTC m=+1853.383295557" watchObservedRunningTime="2026-09-09 16:22:28.517755489 +0000 UTC m=+1853.384266709"
Sep 09 16:22:39.496793 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:39.496756    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"
Sep 09 16:22:39.497275 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:39.496850    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"
Sep 09 16:22:50.901221 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:50.901185    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:22:56.078741 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:22:56.078708    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m_3fcd01e5-8368-471b-9586-5454464f5c81/payload-processing/0.log"
Sep 09 16:23:03.392498 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:03.392397    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:23:04.734350 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.734319    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"]
Sep 09 16:23:04.737887 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.737868    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.741260 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.741238    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8cc2d6\"/\"kube-root-ca.crt\""
Sep 09 16:23:04.741398 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.741263    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8cc2d6\"/\"ipp-route-8cc2d6-kserve-self-signed-certs\""
Sep 09 16:23:04.741398 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.741279    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8cc2d6\"/\"default-dockercfg-jdtxg\""
Sep 09 16:23:04.741398 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.741282    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8cc2d6\"/\"openshift-service-ca.crt\""
Sep 09 16:23:04.748001 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.747945    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"]
Sep 09 16:23:04.760701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.760670    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:23:04.843410 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.843374    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-model-cache\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.843410 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.843419    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f946b3f-c439-4a8d-aa85-ca1af47677f1-tls-certs\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.843684 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.843437    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-dshm\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.843684 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.843505    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsr65\" (UniqueName: \"kubernetes.io/projected/6f946b3f-c439-4a8d-aa85-ca1af47677f1-kube-api-access-hsr65\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.843684 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.843548    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-home\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.843684 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.843606    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-tmp-dir\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.944531 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.944483    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-model-cache\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.944695 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.944544    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f946b3f-c439-4a8d-aa85-ca1af47677f1-tls-certs\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.944695 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.944570    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-dshm\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.944695 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.944600    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsr65\" (UniqueName: \"kubernetes.io/projected/6f946b3f-c439-4a8d-aa85-ca1af47677f1-kube-api-access-hsr65\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.944695 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.944629    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-home\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.944695 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.944682    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-tmp-dir\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.945000 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.944977    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-model-cache\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.945067 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.945007    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-tmp-dir\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.945122 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.945091    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-home\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.946801 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.946775    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-dshm\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.947059 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.947044    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f946b3f-c439-4a8d-aa85-ca1af47677f1-tls-certs\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:04.953761 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:04.953736    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsr65\" (UniqueName: \"kubernetes.io/projected/6f946b3f-c439-4a8d-aa85-ca1af47677f1-kube-api-access-hsr65\") pod \"ipp-route-8cc2d6-kserve-8544b9db7-5hk25\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") " pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:05.048149 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:05.048064    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:05.326047 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:05.325966    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"]
Sep 09 16:23:05.329359 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:23:05.329329    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f946b3f_c439_4a8d_aa85_ca1af47677f1.slice/crio-34f9b81fa850e86eec859ce54a745ed647b00c562dcfacfbc74c6e0409380ceb WatchSource:0}: Error finding container 34f9b81fa850e86eec859ce54a745ed647b00c562dcfacfbc74c6e0409380ceb: Status 404 returned error can't find the container with id 34f9b81fa850e86eec859ce54a745ed647b00c562dcfacfbc74c6e0409380ceb
Sep 09 16:23:05.663029 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:05.662938    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25" event={"ID":"6f946b3f-c439-4a8d-aa85-ca1af47677f1","Type":"ContainerStarted","Data":"2d095052dbf0cbaeae143298a11c753ee910489f1f22329be8a4692e38c39162"}
Sep 09 16:23:05.663029 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:05.662986    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25" event={"ID":"6f946b3f-c439-4a8d-aa85-ca1af47677f1","Type":"ContainerStarted","Data":"34f9b81fa850e86eec859ce54a745ed647b00c562dcfacfbc74c6e0409380ceb"}
Sep 09 16:23:05.685283 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:05.685219    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25" podStartSLOduration=1.685202748 podStartE2EDuration="1.685202748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:23:05.683374239 +0000 UTC m=+1890.549885460" watchObservedRunningTime="2026-09-09 16:23:05.685202748 +0000 UTC m=+1890.551714037"
Sep 09 16:23:15.048716 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:15.048683    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:15.049210 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:15.048909    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:15.061436 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:15.061406    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:15.720871 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:15.720836    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:23:16.250479 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:16.250428    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:23:16.351525 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:16.351494    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:23:37.025995 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:37.025960    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m_3fcd01e5-8368-471b-9586-5454464f5c81/payload-processing/0.log"
Sep 09 16:23:37.291757 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:37.291669    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:23:37.602581 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:37.602495    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:23:37.954319 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:37.954281    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:23:40.225670 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:40.225630    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:23:40.329621 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:40.329587    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:23:41.400552 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:41.400519    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:23:41.502534 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:41.502489    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:23:41.601855 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:41.601822    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:23:41.915902 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:41.915864    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:23:42.016417 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.016379    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:23:42.023594 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.023568    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"]
Sep 09 16:23:42.026864 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.026844    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"
Sep 09 16:23:42.029202 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.029183    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-6ddd57\""
Sep 09 16:23:42.029316 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.029295    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-6ddd57-dockercfg-xszgw\""
Sep 09 16:23:42.038140 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.038112    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"]
Sep 09 16:23:42.067147 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.067113    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm8sx\" (UniqueName: \"kubernetes.io/projected/1dbff579-9e37-4e6d-91cf-db21a86a6469-kube-api-access-bm8sx\") pod \"payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5\" (UID: \"1dbff579-9e37-4e6d-91cf-db21a86a6469\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"
Sep 09 16:23:42.067332 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.067278    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbff579-9e37-4e6d-91cf-db21a86a6469-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5\" (UID: \"1dbff579-9e37-4e6d-91cf-db21a86a6469\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"
Sep 09 16:23:42.168701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.168610    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbff579-9e37-4e6d-91cf-db21a86a6469-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5\" (UID: \"1dbff579-9e37-4e6d-91cf-db21a86a6469\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"
Sep 09 16:23:42.168701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.168656    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm8sx\" (UniqueName: \"kubernetes.io/projected/1dbff579-9e37-4e6d-91cf-db21a86a6469-kube-api-access-bm8sx\") pod \"payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5\" (UID: \"1dbff579-9e37-4e6d-91cf-db21a86a6469\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"
Sep 09 16:23:42.169315 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.169295    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbff579-9e37-4e6d-91cf-db21a86a6469-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5\" (UID: \"1dbff579-9e37-4e6d-91cf-db21a86a6469\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"
Sep 09 16:23:42.176655 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.176621    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm8sx\" (UniqueName: \"kubernetes.io/projected/1dbff579-9e37-4e6d-91cf-db21a86a6469-kube-api-access-bm8sx\") pod \"payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5\" (UID: \"1dbff579-9e37-4e6d-91cf-db21a86a6469\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"
Sep 09 16:23:42.337041 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.337000    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"
Sep 09 16:23:42.606638 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.606552    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"]
Sep 09 16:23:42.608973 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:23:42.608941    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbff579_9e37_4e6d_91cf_db21a86a6469.slice/crio-2750bb9d26ef54976ae4fdfd7d54e833b81d017835b460659b23cc9dbb4384fe WatchSource:0}: Error finding container 2750bb9d26ef54976ae4fdfd7d54e833b81d017835b460659b23cc9dbb4384fe: Status 404 returned error can't find the container with id 2750bb9d26ef54976ae4fdfd7d54e833b81d017835b460659b23cc9dbb4384fe
Sep 09 16:23:42.838749 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.838715    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5" event={"ID":"1dbff579-9e37-4e6d-91cf-db21a86a6469","Type":"ContainerStarted","Data":"8dc6150fb470d44156c88085fe6d816c262c4f64464b033b15adc4677673a48c"}
Sep 09 16:23:42.838917 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.838771    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5" event={"ID":"1dbff579-9e37-4e6d-91cf-db21a86a6469","Type":"ContainerStarted","Data":"2750bb9d26ef54976ae4fdfd7d54e833b81d017835b460659b23cc9dbb4384fe"}
Sep 09 16:23:42.838917 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.838861    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"
Sep 09 16:23:42.855458 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:42.855400    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5" podStartSLOduration=0.855384381 podStartE2EDuration="855.384381ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:23:42.854192915 +0000 UTC m=+1927.720704149" watchObservedRunningTime="2026-09-09 16:23:42.855384381 +0000 UTC m=+1927.721895601"
Sep 09 16:23:53.846924 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:23:53.846893    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"
Sep 09 16:24:05.272145 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:05.272090    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:06.012759 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:06.012722    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q"]
Sep 09 16:24:06.016040 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:06.016023    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q"
Sep 09 16:24:06.018717 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:06.018696    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:24:06.018823 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:06.018732    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\""
Sep 09 16:24:06.024800 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:06.024777    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q"]
Sep 09 16:24:06.190718 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:06.190684    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06f221b-6170-4c3e-ad22-a01623430ea4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q\" (UID: \"d06f221b-6170-4c3e-ad22-a01623430ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q"
Sep 09 16:24:06.291958 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:06.291865    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06f221b-6170-4c3e-ad22-a01623430ea4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q\" (UID: \"d06f221b-6170-4c3e-ad22-a01623430ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q"
Sep 09 16:24:06.292564 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:06.292544    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06f221b-6170-4c3e-ad22-a01623430ea4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q\" (UID: \"d06f221b-6170-4c3e-ad22-a01623430ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q"
Sep 09 16:24:06.327633 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:06.327600    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q"
Sep 09 16:24:06.607129 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:06.607103    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q"]
Sep 09 16:24:06.608840 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:24:06.608812    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06f221b_6170_4c3e_ad22_a01623430ea4.slice/crio-5ee019ebf2509916f1025cddc439bc0a6ee0332dd6106241f247fd6a38bf46d7 WatchSource:0}: Error finding container 5ee019ebf2509916f1025cddc439bc0a6ee0332dd6106241f247fd6a38bf46d7: Status 404 returned error can't find the container with id 5ee019ebf2509916f1025cddc439bc0a6ee0332dd6106241f247fd6a38bf46d7
Sep 09 16:24:06.955789 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:06.955754    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q" event={"ID":"d06f221b-6170-4c3e-ad22-a01623430ea4","Type":"ContainerStarted","Data":"975e8b5e2025f657213203b210f0fb45230f40664db60db412b596cedfed9500"}
Sep 09 16:24:06.955935 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:06.955808    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q" event={"ID":"d06f221b-6170-4c3e-ad22-a01623430ea4","Type":"ContainerStarted","Data":"5ee019ebf2509916f1025cddc439bc0a6ee0332dd6106241f247fd6a38bf46d7"}
Sep 09 16:24:06.970873 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:06.970823    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q" podStartSLOduration=1.97080235 podStartE2EDuration="1.97080235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:24:06.969794449 +0000 UTC m=+1951.836305683" watchObservedRunningTime="2026-09-09 16:24:06.97080235 +0000 UTC m=+1951.837313571"
Sep 09 16:24:07.962123 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:07.962089    2556 generic.go:356] "Generic (PLEG): container finished" podID="d06f221b-6170-4c3e-ad22-a01623430ea4" containerID="975e8b5e2025f657213203b210f0fb45230f40664db60db412b596cedfed9500" exitCode=0
Sep 09 16:24:07.962553 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:07.962162    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q" event={"ID":"d06f221b-6170-4c3e-ad22-a01623430ea4","Type":"ContainerDied","Data":"975e8b5e2025f657213203b210f0fb45230f40664db60db412b596cedfed9500"}
Sep 09 16:24:09.094185 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:09.094162    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q"
Sep 09 16:24:09.218139 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:09.218046    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06f221b-6170-4c3e-ad22-a01623430ea4-maas-api-service-ca\") pod \"d06f221b-6170-4c3e-ad22-a01623430ea4\" (UID: \"d06f221b-6170-4c3e-ad22-a01623430ea4\") "
Sep 09 16:24:09.218448 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:09.218428    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06f221b-6170-4c3e-ad22-a01623430ea4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d06f221b-6170-4c3e-ad22-a01623430ea4" (UID: "d06f221b-6170-4c3e-ad22-a01623430ea4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:24:09.319649 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:09.319612    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06f221b-6170-4c3e-ad22-a01623430ea4-maas-api-service-ca\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:24:09.973353 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:09.973322    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q" event={"ID":"d06f221b-6170-4c3e-ad22-a01623430ea4","Type":"ContainerDied","Data":"5ee019ebf2509916f1025cddc439bc0a6ee0332dd6106241f247fd6a38bf46d7"}
Sep 09 16:24:09.973353 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:09.973345    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q"
Sep 09 16:24:09.973579 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:09.973375    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ee019ebf2509916f1025cddc439bc0a6ee0332dd6106241f247fd6a38bf46d7"
Sep 09 16:24:12.034204 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.034170    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:12.049417 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.049383    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"]
Sep 09 16:24:12.049663 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.049637    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5" podUID="1dbff579-9e37-4e6d-91cf-db21a86a6469" containerName="payload-pre-processing" containerID="cri-o://8dc6150fb470d44156c88085fe6d816c262c4f64464b033b15adc4677673a48c" gracePeriod=30
Sep 09 16:24:12.134256 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.134207    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:12.238359 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.238252    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q"]
Sep 09 16:24:12.240373 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.240345    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6ddd57-7jx7q"]
Sep 09 16:24:12.245941 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.245918    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:12.304548 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.304523    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"
Sep 09 16:24:12.343385 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.343350    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:12.449543 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.449506    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm8sx\" (UniqueName: \"kubernetes.io/projected/1dbff579-9e37-4e6d-91cf-db21a86a6469-kube-api-access-bm8sx\") pod \"1dbff579-9e37-4e6d-91cf-db21a86a6469\" (UID: \"1dbff579-9e37-4e6d-91cf-db21a86a6469\") "
Sep 09 16:24:12.449726 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.449555    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbff579-9e37-4e6d-91cf-db21a86a6469-plugins-config-volume\") pod \"1dbff579-9e37-4e6d-91cf-db21a86a6469\" (UID: \"1dbff579-9e37-4e6d-91cf-db21a86a6469\") "
Sep 09 16:24:12.449944 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.449920    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbff579-9e37-4e6d-91cf-db21a86a6469-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1dbff579-9e37-4e6d-91cf-db21a86a6469" (UID: "1dbff579-9e37-4e6d-91cf-db21a86a6469"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:24:12.451712 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.451686    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbff579-9e37-4e6d-91cf-db21a86a6469-kube-api-access-bm8sx" (OuterVolumeSpecName: "kube-api-access-bm8sx") pod "1dbff579-9e37-4e6d-91cf-db21a86a6469" (UID: "1dbff579-9e37-4e6d-91cf-db21a86a6469"). InnerVolumeSpecName "kube-api-access-bm8sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:24:12.550450 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.550364    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm8sx\" (UniqueName: \"kubernetes.io/projected/1dbff579-9e37-4e6d-91cf-db21a86a6469-kube-api-access-bm8sx\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:24:12.550450 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.550395    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbff579-9e37-4e6d-91cf-db21a86a6469-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:24:12.661822 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.661784    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:12.988057 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.988023    2556 generic.go:356] "Generic (PLEG): container finished" podID="1dbff579-9e37-4e6d-91cf-db21a86a6469" containerID="8dc6150fb470d44156c88085fe6d816c262c4f64464b033b15adc4677673a48c" exitCode=0
Sep 09 16:24:12.988218 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.988068    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"
Sep 09 16:24:12.988218 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.988075    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5" event={"ID":"1dbff579-9e37-4e6d-91cf-db21a86a6469","Type":"ContainerDied","Data":"8dc6150fb470d44156c88085fe6d816c262c4f64464b033b15adc4677673a48c"}
Sep 09 16:24:12.988218 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.988118    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5" event={"ID":"1dbff579-9e37-4e6d-91cf-db21a86a6469","Type":"ContainerDied","Data":"2750bb9d26ef54976ae4fdfd7d54e833b81d017835b460659b23cc9dbb4384fe"}
Sep 09 16:24:12.988218 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.988144    2556 scope.go:117] "RemoveContainer" containerID="8dc6150fb470d44156c88085fe6d816c262c4f64464b033b15adc4677673a48c"
Sep 09 16:24:12.996874 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.996853    2556 scope.go:117] "RemoveContainer" containerID="8dc6150fb470d44156c88085fe6d816c262c4f64464b033b15adc4677673a48c"
Sep 09 16:24:12.997155 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:24:12.997135    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc6150fb470d44156c88085fe6d816c262c4f64464b033b15adc4677673a48c\": container with ID starting with 8dc6150fb470d44156c88085fe6d816c262c4f64464b033b15adc4677673a48c not found: ID does not exist" containerID="8dc6150fb470d44156c88085fe6d816c262c4f64464b033b15adc4677673a48c"
Sep 09 16:24:12.997209 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:12.997164    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc6150fb470d44156c88085fe6d816c262c4f64464b033b15adc4677673a48c"} err="failed to get container status \"8dc6150fb470d44156c88085fe6d816c262c4f64464b033b15adc4677673a48c\": rpc error: code = NotFound desc = could not find container \"8dc6150fb470d44156c88085fe6d816c262c4f64464b033b15adc4677673a48c\": container with ID starting with 8dc6150fb470d44156c88085fe6d816c262c4f64464b033b15adc4677673a48c not found: ID does not exist"
Sep 09 16:24:13.008479 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:13.008441    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"]
Sep 09 16:24:13.012398 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:13.012379    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6ddd57-65fd6c5746-t8hx5"]
Sep 09 16:24:13.763374 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:13.763342    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dbff579-9e37-4e6d-91cf-db21a86a6469" path="/var/lib/kubelet/pods/1dbff579-9e37-4e6d-91cf-db21a86a6469/volumes"
Sep 09 16:24:13.763775 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:13.763676    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d06f221b-6170-4c3e-ad22-a01623430ea4" path="/var/lib/kubelet/pods/d06f221b-6170-4c3e-ad22-a01623430ea4/volumes"
Sep 09 16:24:18.297906 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:18.297871    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:18.397763 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:18.397730    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:18.581851 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:18.581765    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:18.680863 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:18.680831    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:22.211990 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.211955    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg"]
Sep 09 16:24:22.212379 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.212335    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dbff579-9e37-4e6d-91cf-db21a86a6469" containerName="payload-pre-processing"
Sep 09 16:24:22.212379 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.212347    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dbff579-9e37-4e6d-91cf-db21a86a6469" containerName="payload-pre-processing"
Sep 09 16:24:22.212379 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.212359    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d06f221b-6170-4c3e-ad22-a01623430ea4" containerName="revoke-keys"
Sep 09 16:24:22.212379 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.212365    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d06f221b-6170-4c3e-ad22-a01623430ea4" containerName="revoke-keys"
Sep 09 16:24:22.212535 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.212416    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dbff579-9e37-4e6d-91cf-db21a86a6469" containerName="payload-pre-processing"
Sep 09 16:24:22.212535 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.212424    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d06f221b-6170-4c3e-ad22-a01623430ea4" containerName="revoke-keys"
Sep 09 16:24:22.216833 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.216813    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg"
Sep 09 16:24:22.219215 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.219196    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\""
Sep 09 16:24:22.219418 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.219403    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:24:22.223001 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.222974    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg"]
Sep 09 16:24:22.323672 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.323633    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca58c2c5-1f43-4d8a-8295-3d36277e3cfb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg\" (UID: \"ca58c2c5-1f43-4d8a-8295-3d36277e3cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg"
Sep 09 16:24:22.424586 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.424546    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca58c2c5-1f43-4d8a-8295-3d36277e3cfb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg\" (UID: \"ca58c2c5-1f43-4d8a-8295-3d36277e3cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg"
Sep 09 16:24:22.425134 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.425115    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca58c2c5-1f43-4d8a-8295-3d36277e3cfb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg\" (UID: \"ca58c2c5-1f43-4d8a-8295-3d36277e3cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg"
Sep 09 16:24:22.529573 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.529458    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg"
Sep 09 16:24:22.574625 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.574587    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:22.796442 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:22.796417    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg"]
Sep 09 16:24:22.799185 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:24:22.799161    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca58c2c5_1f43_4d8a_8295_3d36277e3cfb.slice/crio-fe0a642949e010e8f41fd8c05fc907f088b6272ca55771fb0f5468fc06259b86 WatchSource:0}: Error finding container fe0a642949e010e8f41fd8c05fc907f088b6272ca55771fb0f5468fc06259b86: Status 404 returned error can't find the container with id fe0a642949e010e8f41fd8c05fc907f088b6272ca55771fb0f5468fc06259b86
Sep 09 16:24:23.036017 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:23.035992    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg" event={"ID":"ca58c2c5-1f43-4d8a-8295-3d36277e3cfb","Type":"ContainerStarted","Data":"fe0a642949e010e8f41fd8c05fc907f088b6272ca55771fb0f5468fc06259b86"}
Sep 09 16:24:24.042145 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:24.042108    2556 generic.go:356] "Generic (PLEG): container finished" podID="ca58c2c5-1f43-4d8a-8295-3d36277e3cfb" containerID="3b0495de82a6d0a77e4b105fc52aebc57421da1beea35d90269317c5461e4d1b" exitCode=0
Sep 09 16:24:24.042639 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:24.042155    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg" event={"ID":"ca58c2c5-1f43-4d8a-8295-3d36277e3cfb","Type":"ContainerDied","Data":"3b0495de82a6d0a77e4b105fc52aebc57421da1beea35d90269317c5461e4d1b"}
Sep 09 16:24:25.175528 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:25.175506    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg"
Sep 09 16:24:25.250644 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:25.250614    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca58c2c5-1f43-4d8a-8295-3d36277e3cfb-maas-api-service-ca\") pod \"ca58c2c5-1f43-4d8a-8295-3d36277e3cfb\" (UID: \"ca58c2c5-1f43-4d8a-8295-3d36277e3cfb\") "
Sep 09 16:24:25.250933 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:25.250912    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca58c2c5-1f43-4d8a-8295-3d36277e3cfb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca58c2c5-1f43-4d8a-8295-3d36277e3cfb" (UID: "ca58c2c5-1f43-4d8a-8295-3d36277e3cfb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:24:25.352110 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:25.352027    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca58c2c5-1f43-4d8a-8295-3d36277e3cfb-maas-api-service-ca\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:24:26.052477 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:26.052373    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg" event={"ID":"ca58c2c5-1f43-4d8a-8295-3d36277e3cfb","Type":"ContainerDied","Data":"fe0a642949e010e8f41fd8c05fc907f088b6272ca55771fb0f5468fc06259b86"}
Sep 09 16:24:26.052477 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:26.052425    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe0a642949e010e8f41fd8c05fc907f088b6272ca55771fb0f5468fc06259b86"
Sep 09 16:24:26.052477 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:26.052386    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg"
Sep 09 16:24:31.534382 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:31.534291    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:32.358480 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:32.358428    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:37.951268 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:37.951228    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:37.977046 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:37.977011    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"]
Sep 09 16:24:37.977293 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:37.977267    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk" podUID="ac7efd0e-8b7f-4a46-8203-a268ec09d940" containerName="payload-pre-processing" containerID="cri-o://552b55088f2ba33ec4ae7b952e8b37b6bdb8249548f4066128bfd406a3d26221" gracePeriod=30
Sep 09 16:24:38.052564 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:38.052526    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:38.112063 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:38.112033    2556 generic.go:356] "Generic (PLEG): container finished" podID="ac7efd0e-8b7f-4a46-8203-a268ec09d940" containerID="552b55088f2ba33ec4ae7b952e8b37b6bdb8249548f4066128bfd406a3d26221" exitCode=0
Sep 09 16:24:38.112219 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:38.112103    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk" event={"ID":"ac7efd0e-8b7f-4a46-8203-a268ec09d940","Type":"ContainerDied","Data":"552b55088f2ba33ec4ae7b952e8b37b6bdb8249548f4066128bfd406a3d26221"}
Sep 09 16:24:38.148710 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:38.148674    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg"]
Sep 09 16:24:38.151306 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:38.151278    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8cc2d6-pwjqg"]
Sep 09 16:24:38.151973 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:38.151947    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:38.218539 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:38.218509    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:38.229942 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:38.229921    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"
Sep 09 16:24:38.360986 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:38.360952    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lk4z\" (UniqueName: \"kubernetes.io/projected/ac7efd0e-8b7f-4a46-8203-a268ec09d940-kube-api-access-5lk4z\") pod \"ac7efd0e-8b7f-4a46-8203-a268ec09d940\" (UID: \"ac7efd0e-8b7f-4a46-8203-a268ec09d940\") "
Sep 09 16:24:38.360986 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:38.360988    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7efd0e-8b7f-4a46-8203-a268ec09d940-plugins-config-volume\") pod \"ac7efd0e-8b7f-4a46-8203-a268ec09d940\" (UID: \"ac7efd0e-8b7f-4a46-8203-a268ec09d940\") "
Sep 09 16:24:38.361454 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:38.361427    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7efd0e-8b7f-4a46-8203-a268ec09d940-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac7efd0e-8b7f-4a46-8203-a268ec09d940" (UID: "ac7efd0e-8b7f-4a46-8203-a268ec09d940"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:24:38.363066 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:38.363041    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7efd0e-8b7f-4a46-8203-a268ec09d940-kube-api-access-5lk4z" (OuterVolumeSpecName: "kube-api-access-5lk4z") pod "ac7efd0e-8b7f-4a46-8203-a268ec09d940" (UID: "ac7efd0e-8b7f-4a46-8203-a268ec09d940"). InnerVolumeSpecName "kube-api-access-5lk4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:24:38.461739 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:38.461644    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lk4z\" (UniqueName: \"kubernetes.io/projected/ac7efd0e-8b7f-4a46-8203-a268ec09d940-kube-api-access-5lk4z\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:24:38.461739 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:38.461678    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7efd0e-8b7f-4a46-8203-a268ec09d940-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:24:38.531445 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:38.531408    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:39.117722 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:39.117693    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"
Sep 09 16:24:39.117722 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:39.117714    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk" event={"ID":"ac7efd0e-8b7f-4a46-8203-a268ec09d940","Type":"ContainerDied","Data":"f0622b5a1e2cb8ebd0c4923b1db8a11879152c5a72ea9bf16f8767792252377a"}
Sep 09 16:24:39.118201 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:39.117780    2556 scope.go:117] "RemoveContainer" containerID="552b55088f2ba33ec4ae7b952e8b37b6bdb8249548f4066128bfd406a3d26221"
Sep 09 16:24:39.138249 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:39.138226    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"]
Sep 09 16:24:39.141876 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:39.141852    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8cc2d6-68f89c5558-j74kk"]
Sep 09 16:24:39.763421 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:39.763383    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac7efd0e-8b7f-4a46-8203-a268ec09d940" path="/var/lib/kubelet/pods/ac7efd0e-8b7f-4a46-8203-a268ec09d940/volumes"
Sep 09 16:24:39.763779 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:39.763764    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca58c2c5-1f43-4d8a-8295-3d36277e3cfb" path="/var/lib/kubelet/pods/ca58c2c5-1f43-4d8a-8295-3d36277e3cfb/volumes"
Sep 09 16:24:43.507401 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:43.507363    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"]
Sep 09 16:24:43.550090 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:43.550061    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:48.596915 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:48.596875    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:58.526953 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:58.526888    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25" podUID="6f946b3f-c439-4a8d-aa85-ca1af47677f1" containerName="main" containerID="cri-o://2d095052dbf0cbaeae143298a11c753ee910489f1f22329be8a4692e38c39162" gracePeriod=45
Sep 09 16:24:58.767333 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:58.767308    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:24:58.937512 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:58.937483    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsr65\" (UniqueName: \"kubernetes.io/projected/6f946b3f-c439-4a8d-aa85-ca1af47677f1-kube-api-access-hsr65\") pod \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") "
Sep 09 16:24:58.937678 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:58.937541    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-dshm\") pod \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") "
Sep 09 16:24:58.937678 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:58.937596    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-tmp-dir\") pod \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") "
Sep 09 16:24:58.937678 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:58.937644    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f946b3f-c439-4a8d-aa85-ca1af47677f1-tls-certs\") pod \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") "
Sep 09 16:24:58.937678 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:58.937659    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-home\") pod \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") "
Sep 09 16:24:58.937678 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:58.937675    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-model-cache\") pod \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\" (UID: \"6f946b3f-c439-4a8d-aa85-ca1af47677f1\") "
Sep 09 16:24:58.938008 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:58.937931    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f946b3f-c439-4a8d-aa85-ca1af47677f1" (UID: "6f946b3f-c439-4a8d-aa85-ca1af47677f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:24:58.938008 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:58.937956    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-home" (OuterVolumeSpecName: "home") pod "6f946b3f-c439-4a8d-aa85-ca1af47677f1" (UID: "6f946b3f-c439-4a8d-aa85-ca1af47677f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:24:58.938008 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:58.938000    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f946b3f-c439-4a8d-aa85-ca1af47677f1" (UID: "6f946b3f-c439-4a8d-aa85-ca1af47677f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:24:58.939757 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:58.939729    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-dshm" (OuterVolumeSpecName: "dshm") pod "6f946b3f-c439-4a8d-aa85-ca1af47677f1" (UID: "6f946b3f-c439-4a8d-aa85-ca1af47677f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:24:58.939841 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:58.939793    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f946b3f-c439-4a8d-aa85-ca1af47677f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f946b3f-c439-4a8d-aa85-ca1af47677f1" (UID: "6f946b3f-c439-4a8d-aa85-ca1af47677f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:24:58.939841 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:58.939797    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f946b3f-c439-4a8d-aa85-ca1af47677f1-kube-api-access-hsr65" (OuterVolumeSpecName: "kube-api-access-hsr65") pod "6f946b3f-c439-4a8d-aa85-ca1af47677f1" (UID: "6f946b3f-c439-4a8d-aa85-ca1af47677f1"). InnerVolumeSpecName "kube-api-access-hsr65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:24:59.038883 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.038852    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-tmp-dir\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:24:59.038883 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.038877    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f946b3f-c439-4a8d-aa85-ca1af47677f1-tls-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:24:59.039065 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.038892    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-home\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:24:59.039065 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.038903    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-model-cache\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:24:59.039065 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.038912    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsr65\" (UniqueName: \"kubernetes.io/projected/6f946b3f-c439-4a8d-aa85-ca1af47677f1-kube-api-access-hsr65\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:24:59.039065 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.038920    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f946b3f-c439-4a8d-aa85-ca1af47677f1-dshm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:24:59.218396 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.218300    2556 generic.go:356] "Generic (PLEG): container finished" podID="6f946b3f-c439-4a8d-aa85-ca1af47677f1" containerID="2d095052dbf0cbaeae143298a11c753ee910489f1f22329be8a4692e38c39162" exitCode=0
Sep 09 16:24:59.218396 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.218346    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"
Sep 09 16:24:59.218396 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.218370    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25" event={"ID":"6f946b3f-c439-4a8d-aa85-ca1af47677f1","Type":"ContainerDied","Data":"2d095052dbf0cbaeae143298a11c753ee910489f1f22329be8a4692e38c39162"}
Sep 09 16:24:59.218680 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.218408    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25" event={"ID":"6f946b3f-c439-4a8d-aa85-ca1af47677f1","Type":"ContainerDied","Data":"34f9b81fa850e86eec859ce54a745ed647b00c562dcfacfbc74c6e0409380ceb"}
Sep 09 16:24:59.218680 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.218430    2556 scope.go:117] "RemoveContainer" containerID="2d095052dbf0cbaeae143298a11c753ee910489f1f22329be8a4692e38c39162"
Sep 09 16:24:59.227217 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.227197    2556 scope.go:117] "RemoveContainer" containerID="2d095052dbf0cbaeae143298a11c753ee910489f1f22329be8a4692e38c39162"
Sep 09 16:24:59.227500 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:24:59.227477    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d095052dbf0cbaeae143298a11c753ee910489f1f22329be8a4692e38c39162\": container with ID starting with 2d095052dbf0cbaeae143298a11c753ee910489f1f22329be8a4692e38c39162 not found: ID does not exist" containerID="2d095052dbf0cbaeae143298a11c753ee910489f1f22329be8a4692e38c39162"
Sep 09 16:24:59.227595 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.227509    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d095052dbf0cbaeae143298a11c753ee910489f1f22329be8a4692e38c39162"} err="failed to get container status \"2d095052dbf0cbaeae143298a11c753ee910489f1f22329be8a4692e38c39162\": rpc error: code = NotFound desc = could not find container \"2d095052dbf0cbaeae143298a11c753ee910489f1f22329be8a4692e38c39162\": container with ID starting with 2d095052dbf0cbaeae143298a11c753ee910489f1f22329be8a4692e38c39162 not found: ID does not exist"
Sep 09 16:24:59.239594 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.239569    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"]
Sep 09 16:24:59.241121 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.241101    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-8cc2d6/ipp-route-8cc2d6-kserve-8544b9db7-5hk25"]
Sep 09 16:24:59.743900 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.743864    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:24:59.763221 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:24:59.763192    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f946b3f-c439-4a8d-aa85-ca1af47677f1" path="/var/lib/kubelet/pods/6f946b3f-c439-4a8d-aa85-ca1af47677f1/volumes"
Sep 09 16:25:20.227554 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:20.227515    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:20.493012 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:20.492919    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:26.131255 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.131222    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:26.265392 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.265358    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:26.414954 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.414868    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:26.770593 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.770561    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d"]
Sep 09 16:25:26.770913 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.770898    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac7efd0e-8b7f-4a46-8203-a268ec09d940" containerName="payload-pre-processing"
Sep 09 16:25:26.770913 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.770911    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac7efd0e-8b7f-4a46-8203-a268ec09d940" containerName="payload-pre-processing"
Sep 09 16:25:26.771051 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.770928    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca58c2c5-1f43-4d8a-8295-3d36277e3cfb" containerName="revoke-keys"
Sep 09 16:25:26.771051 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.770933    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca58c2c5-1f43-4d8a-8295-3d36277e3cfb" containerName="revoke-keys"
Sep 09 16:25:26.771051 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.770941    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f946b3f-c439-4a8d-aa85-ca1af47677f1" containerName="main"
Sep 09 16:25:26.771051 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.770946    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f946b3f-c439-4a8d-aa85-ca1af47677f1" containerName="main"
Sep 09 16:25:26.771051 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.771004    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca58c2c5-1f43-4d8a-8295-3d36277e3cfb" containerName="revoke-keys"
Sep 09 16:25:26.771051 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.771012    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f946b3f-c439-4a8d-aa85-ca1af47677f1" containerName="main"
Sep 09 16:25:26.771051 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.771020    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac7efd0e-8b7f-4a46-8203-a268ec09d940" containerName="payload-pre-processing"
Sep 09 16:25:26.773916 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.773900    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d"
Sep 09 16:25:26.776196 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.776177    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:25:26.776320 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.776304    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\""
Sep 09 16:25:26.781348 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.781168    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d"]
Sep 09 16:25:26.862537 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.862496    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a89aed-96a2-46d8-83ed-e6a6956efb4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d\" (UID: \"08a89aed-96a2-46d8-83ed-e6a6956efb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d"
Sep 09 16:25:26.945302 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.945272    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:26.963365 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.963336    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a89aed-96a2-46d8-83ed-e6a6956efb4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d\" (UID: \"08a89aed-96a2-46d8-83ed-e6a6956efb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d"
Sep 09 16:25:26.963925 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:26.963907    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a89aed-96a2-46d8-83ed-e6a6956efb4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d\" (UID: \"08a89aed-96a2-46d8-83ed-e6a6956efb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d"
Sep 09 16:25:27.085518 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:27.085418    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d"
Sep 09 16:25:27.356841 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:27.356812    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d"]
Sep 09 16:25:28.355044 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:28.354956    2556 generic.go:356] "Generic (PLEG): container finished" podID="08a89aed-96a2-46d8-83ed-e6a6956efb4f" containerID="bb4fd02fc3ff718c4329e98f4ccb4f9f36708f82f92a5f4c6e06f7eda016a92a" exitCode=0
Sep 09 16:25:28.355196 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:28.355043    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d" event={"ID":"08a89aed-96a2-46d8-83ed-e6a6956efb4f","Type":"ContainerDied","Data":"bb4fd02fc3ff718c4329e98f4ccb4f9f36708f82f92a5f4c6e06f7eda016a92a"}
Sep 09 16:25:28.355196 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:28.355094    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d" event={"ID":"08a89aed-96a2-46d8-83ed-e6a6956efb4f","Type":"ContainerStarted","Data":"07eeb40ba837c77714b1fad46265a5c730172b5399a49a51cbaee52986ff3822"}
Sep 09 16:25:29.485558 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:29.485534    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d"
Sep 09 16:25:29.586365 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:29.586328    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a89aed-96a2-46d8-83ed-e6a6956efb4f-maas-api-service-ca\") pod \"08a89aed-96a2-46d8-83ed-e6a6956efb4f\" (UID: \"08a89aed-96a2-46d8-83ed-e6a6956efb4f\") "
Sep 09 16:25:29.586764 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:29.586735    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a89aed-96a2-46d8-83ed-e6a6956efb4f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08a89aed-96a2-46d8-83ed-e6a6956efb4f" (UID: "08a89aed-96a2-46d8-83ed-e6a6956efb4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:25:29.687413 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:29.687380    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a89aed-96a2-46d8-83ed-e6a6956efb4f-maas-api-service-ca\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:25:30.365531 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:30.365490    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d" event={"ID":"08a89aed-96a2-46d8-83ed-e6a6956efb4f","Type":"ContainerDied","Data":"07eeb40ba837c77714b1fad46265a5c730172b5399a49a51cbaee52986ff3822"}
Sep 09 16:25:30.365531 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:30.365524    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d"
Sep 09 16:25:30.365748 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:30.365545    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07eeb40ba837c77714b1fad46265a5c730172b5399a49a51cbaee52986ff3822"
Sep 09 16:25:37.259662 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.259623    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:37.275547 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.275519    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"]
Sep 09 16:25:37.275810 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.275784    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m" podUID="3fcd01e5-8368-471b-9586-5454464f5c81" containerName="payload-processing" containerID="cri-o://bed53938fa8cac1e74940157b411f66b054338cc7fcb92aa0de2209aae3c054a" gracePeriod=30
Sep 09 16:25:37.284201 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.284178    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"]
Sep 09 16:25:37.284402 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.284364    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd" podUID="d3662e0e-d5c5-4f54-96fa-e1da62ba781a" containerName="payload-pre-processing" containerID="cri-o://6e6651e31abb91f159cba7ec5e16225e804e106fc76a9b3393f368b37b6f2c72" gracePeriod=30
Sep 09 16:25:37.361397 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.361371    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:37.400360 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.400317    2556 generic.go:356] "Generic (PLEG): container finished" podID="3fcd01e5-8368-471b-9586-5454464f5c81" containerID="bed53938fa8cac1e74940157b411f66b054338cc7fcb92aa0de2209aae3c054a" exitCode=0
Sep 09 16:25:37.400518 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.400387    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m" event={"ID":"3fcd01e5-8368-471b-9586-5454464f5c81","Type":"ContainerDied","Data":"bed53938fa8cac1e74940157b411f66b054338cc7fcb92aa0de2209aae3c054a"}
Sep 09 16:25:37.448007 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.447935    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d"]
Sep 09 16:25:37.454214 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.454176    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:37.461673 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.461086    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b99260-s5l8d"]
Sep 09 16:25:37.515928 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.515893    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:37.532440 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.532417    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"
Sep 09 16:25:37.560260 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.560237    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"
Sep 09 16:25:37.658714 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.658682    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ndkz\" (UniqueName: \"kubernetes.io/projected/d3662e0e-d5c5-4f54-96fa-e1da62ba781a-kube-api-access-9ndkz\") pod \"d3662e0e-d5c5-4f54-96fa-e1da62ba781a\" (UID: \"d3662e0e-d5c5-4f54-96fa-e1da62ba781a\") "
Sep 09 16:25:37.658908 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.658737    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fcd01e5-8368-471b-9586-5454464f5c81-plugins-config-volume\") pod \"3fcd01e5-8368-471b-9586-5454464f5c81\" (UID: \"3fcd01e5-8368-471b-9586-5454464f5c81\") "
Sep 09 16:25:37.658908 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.658779    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3662e0e-d5c5-4f54-96fa-e1da62ba781a-plugins-config-volume\") pod \"d3662e0e-d5c5-4f54-96fa-e1da62ba781a\" (UID: \"d3662e0e-d5c5-4f54-96fa-e1da62ba781a\") "
Sep 09 16:25:37.658908 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.658810    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2fqw\" (UniqueName: \"kubernetes.io/projected/3fcd01e5-8368-471b-9586-5454464f5c81-kube-api-access-d2fqw\") pod \"3fcd01e5-8368-471b-9586-5454464f5c81\" (UID: \"3fcd01e5-8368-471b-9586-5454464f5c81\") "
Sep 09 16:25:37.659217 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.659181    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fcd01e5-8368-471b-9586-5454464f5c81-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3fcd01e5-8368-471b-9586-5454464f5c81" (UID: "3fcd01e5-8368-471b-9586-5454464f5c81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:25:37.659217 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.659203    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3662e0e-d5c5-4f54-96fa-e1da62ba781a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d3662e0e-d5c5-4f54-96fa-e1da62ba781a" (UID: "d3662e0e-d5c5-4f54-96fa-e1da62ba781a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:25:37.660933 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.660908    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fcd01e5-8368-471b-9586-5454464f5c81-kube-api-access-d2fqw" (OuterVolumeSpecName: "kube-api-access-d2fqw") pod "3fcd01e5-8368-471b-9586-5454464f5c81" (UID: "3fcd01e5-8368-471b-9586-5454464f5c81"). InnerVolumeSpecName "kube-api-access-d2fqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:25:37.661005 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.660952    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3662e0e-d5c5-4f54-96fa-e1da62ba781a-kube-api-access-9ndkz" (OuterVolumeSpecName: "kube-api-access-9ndkz") pod "d3662e0e-d5c5-4f54-96fa-e1da62ba781a" (UID: "d3662e0e-d5c5-4f54-96fa-e1da62ba781a"). InnerVolumeSpecName "kube-api-access-9ndkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:25:37.759701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.759667    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3662e0e-d5c5-4f54-96fa-e1da62ba781a-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:25:37.759701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.759693    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2fqw\" (UniqueName: \"kubernetes.io/projected/3fcd01e5-8368-471b-9586-5454464f5c81-kube-api-access-d2fqw\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:25:37.759701 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.759703    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ndkz\" (UniqueName: \"kubernetes.io/projected/d3662e0e-d5c5-4f54-96fa-e1da62ba781a-kube-api-access-9ndkz\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:25:37.759933 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.759712    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fcd01e5-8368-471b-9586-5454464f5c81-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:25:37.763339 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.763278    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08a89aed-96a2-46d8-83ed-e6a6956efb4f" path="/var/lib/kubelet/pods/08a89aed-96a2-46d8-83ed-e6a6956efb4f/volumes"
Sep 09 16:25:37.902368 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:37.902335    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:38.408902 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:38.408872    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"
Sep 09 16:25:38.408902 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:38.408881    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m" event={"ID":"3fcd01e5-8368-471b-9586-5454464f5c81","Type":"ContainerDied","Data":"a0139042f76476feb7b7f3072e8fe76fc4245d5f9eb77e5cc49be6bf05d84b4f"}
Sep 09 16:25:38.409438 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:38.408946    2556 scope.go:117] "RemoveContainer" containerID="bed53938fa8cac1e74940157b411f66b054338cc7fcb92aa0de2209aae3c054a"
Sep 09 16:25:38.410313 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:38.410249    2556 generic.go:356] "Generic (PLEG): container finished" podID="d3662e0e-d5c5-4f54-96fa-e1da62ba781a" containerID="6e6651e31abb91f159cba7ec5e16225e804e106fc76a9b3393f368b37b6f2c72" exitCode=0
Sep 09 16:25:38.410313 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:38.410301    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"
Sep 09 16:25:38.410444 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:38.410330    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd" event={"ID":"d3662e0e-d5c5-4f54-96fa-e1da62ba781a","Type":"ContainerDied","Data":"6e6651e31abb91f159cba7ec5e16225e804e106fc76a9b3393f368b37b6f2c72"}
Sep 09 16:25:38.410444 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:38.410378    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd" event={"ID":"d3662e0e-d5c5-4f54-96fa-e1da62ba781a","Type":"ContainerDied","Data":"815899b3455710d8dad437f1f2e8016c266a68d2f1b869573bb05bb4ab5b052b"}
Sep 09 16:25:38.417636 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:38.417615    2556 scope.go:117] "RemoveContainer" containerID="6e6651e31abb91f159cba7ec5e16225e804e106fc76a9b3393f368b37b6f2c72"
Sep 09 16:25:38.425066 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:38.425048    2556 scope.go:117] "RemoveContainer" containerID="6e6651e31abb91f159cba7ec5e16225e804e106fc76a9b3393f368b37b6f2c72"
Sep 09 16:25:38.425314 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:25:38.425295    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6651e31abb91f159cba7ec5e16225e804e106fc76a9b3393f368b37b6f2c72\": container with ID starting with 6e6651e31abb91f159cba7ec5e16225e804e106fc76a9b3393f368b37b6f2c72 not found: ID does not exist" containerID="6e6651e31abb91f159cba7ec5e16225e804e106fc76a9b3393f368b37b6f2c72"
Sep 09 16:25:38.425374 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:38.425323    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6651e31abb91f159cba7ec5e16225e804e106fc76a9b3393f368b37b6f2c72"} err="failed to get container status \"6e6651e31abb91f159cba7ec5e16225e804e106fc76a9b3393f368b37b6f2c72\": rpc error: code = NotFound desc = could not find container \"6e6651e31abb91f159cba7ec5e16225e804e106fc76a9b3393f368b37b6f2c72\": container with ID starting with 6e6651e31abb91f159cba7ec5e16225e804e106fc76a9b3393f368b37b6f2c72 not found: ID does not exist"
Sep 09 16:25:38.434723 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:38.434701    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"]
Sep 09 16:25:38.448401 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:38.448373    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b99260-bc74f58f5-nxd6m"]
Sep 09 16:25:38.467826 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:38.467798    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"]
Sep 09 16:25:38.469189 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:38.469172    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b99260-5b4bbd77c8-sfbzd"]
Sep 09 16:25:39.763759 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:39.763727    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fcd01e5-8368-471b-9586-5454464f5c81" path="/var/lib/kubelet/pods/3fcd01e5-8368-471b-9586-5454464f5c81/volumes"
Sep 09 16:25:39.764166 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:39.764063    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3662e0e-d5c5-4f54-96fa-e1da62ba781a" path="/var/lib/kubelet/pods/d3662e0e-d5c5-4f54-96fa-e1da62ba781a/volumes"
Sep 09 16:25:43.539476 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:43.539429    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:43.597736 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:43.597697    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:43.800207 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:43.800124    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:44.148196 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.148116    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6"]
Sep 09 16:25:44.148496 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.148481    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3662e0e-d5c5-4f54-96fa-e1da62ba781a" containerName="payload-pre-processing"
Sep 09 16:25:44.148568 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.148500    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3662e0e-d5c5-4f54-96fa-e1da62ba781a" containerName="payload-pre-processing"
Sep 09 16:25:44.148568 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.148519    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fcd01e5-8368-471b-9586-5454464f5c81" containerName="payload-processing"
Sep 09 16:25:44.148568 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.148524    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fcd01e5-8368-471b-9586-5454464f5c81" containerName="payload-processing"
Sep 09 16:25:44.148568 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.148535    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a89aed-96a2-46d8-83ed-e6a6956efb4f" containerName="revoke-keys"
Sep 09 16:25:44.148568 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.148540    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a89aed-96a2-46d8-83ed-e6a6956efb4f" containerName="revoke-keys"
Sep 09 16:25:44.148728 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.148598    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a89aed-96a2-46d8-83ed-e6a6956efb4f" containerName="revoke-keys"
Sep 09 16:25:44.148728 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.148607    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3662e0e-d5c5-4f54-96fa-e1da62ba781a" containerName="payload-pre-processing"
Sep 09 16:25:44.148728 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.148614    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fcd01e5-8368-471b-9586-5454464f5c81" containerName="payload-processing"
Sep 09 16:25:44.151659 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.151644    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6"
Sep 09 16:25:44.153712 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.153688    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\""
Sep 09 16:25:44.153823 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.153688    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:25:44.157830 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.157807    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6"]
Sep 09 16:25:44.213030 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.212990    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f336017a-23c7-4a3e-b94b-37415d5bd4a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6\" (UID: \"f336017a-23c7-4a3e-b94b-37415d5bd4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6"
Sep 09 16:25:44.245677 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.245640    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:44.314044 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.314013    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f336017a-23c7-4a3e-b94b-37415d5bd4a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6\" (UID: \"f336017a-23c7-4a3e-b94b-37415d5bd4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6"
Sep 09 16:25:44.314620 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.314600    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f336017a-23c7-4a3e-b94b-37415d5bd4a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6\" (UID: \"f336017a-23c7-4a3e-b94b-37415d5bd4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6"
Sep 09 16:25:44.462560 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.462517    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6"
Sep 09 16:25:44.730718 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:44.730695    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6"]
Sep 09 16:25:44.733538 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:25:44.733513    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf336017a_23c7_4a3e_b94b_37415d5bd4a7.slice/crio-1d19455eb91d63684a05f25c89dc29cc5321f0a439e2bc117e78588338ce04c0 WatchSource:0}: Error finding container 1d19455eb91d63684a05f25c89dc29cc5321f0a439e2bc117e78588338ce04c0: Status 404 returned error can't find the container with id 1d19455eb91d63684a05f25c89dc29cc5321f0a439e2bc117e78588338ce04c0
Sep 09 16:25:45.445248 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:45.445219    2556 generic.go:356] "Generic (PLEG): container finished" podID="f336017a-23c7-4a3e-b94b-37415d5bd4a7" containerID="c706840b43a97e148b24e7ce51d1f96ca52240bacfb35fc4efc316511e520ac7" exitCode=0
Sep 09 16:25:45.445353 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:45.445303    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6" event={"ID":"f336017a-23c7-4a3e-b94b-37415d5bd4a7","Type":"ContainerDied","Data":"c706840b43a97e148b24e7ce51d1f96ca52240bacfb35fc4efc316511e520ac7"}
Sep 09 16:25:45.445409 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:45.445362    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6" event={"ID":"f336017a-23c7-4a3e-b94b-37415d5bd4a7","Type":"ContainerStarted","Data":"1d19455eb91d63684a05f25c89dc29cc5321f0a439e2bc117e78588338ce04c0"}
Sep 09 16:25:46.576159 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:46.576132    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6"
Sep 09 16:25:46.735252 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:46.735176    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f336017a-23c7-4a3e-b94b-37415d5bd4a7-maas-api-service-ca\") pod \"f336017a-23c7-4a3e-b94b-37415d5bd4a7\" (UID: \"f336017a-23c7-4a3e-b94b-37415d5bd4a7\") "
Sep 09 16:25:46.735514 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:46.735493    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f336017a-23c7-4a3e-b94b-37415d5bd4a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f336017a-23c7-4a3e-b94b-37415d5bd4a7" (UID: "f336017a-23c7-4a3e-b94b-37415d5bd4a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:25:46.836203 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:46.836172    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f336017a-23c7-4a3e-b94b-37415d5bd4a7-maas-api-service-ca\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:25:47.455579 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:47.455544    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6" event={"ID":"f336017a-23c7-4a3e-b94b-37415d5bd4a7","Type":"ContainerDied","Data":"1d19455eb91d63684a05f25c89dc29cc5321f0a439e2bc117e78588338ce04c0"}
Sep 09 16:25:47.455579 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:47.455570    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6"
Sep 09 16:25:47.455801 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:47.455594    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d19455eb91d63684a05f25c89dc29cc5321f0a439e2bc117e78588338ce04c0"
Sep 09 16:25:54.844498 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:54.844447    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"]
Sep 09 16:25:54.844984 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:54.844721    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6" podUID="c843ebc4-ceb1-43a6-a30a-136f5a93140b" containerName="payload-processing" containerID="cri-o://97943b839a74eceb069de6260cf7da32ac9a4014e00d5d2feaf7c50d51eddd71" gracePeriod=30
Sep 09 16:25:54.852628 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:54.852594    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"]
Sep 09 16:25:54.852878 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:54.852851    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn" podUID="2a904a14-212b-42ae-9b1f-ab0ef31a4bac" containerName="payload-pre-processing" containerID="cri-o://4dbff5e7f950993700bbdcffea4ec388b6d76dfccec77f1b3ee476c6194c0432" gracePeriod=30
Sep 09 16:25:55.007725 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.007688    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6"]
Sep 09 16:25:55.011598 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.011567    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-131689-hvkf6"]
Sep 09 16:25:55.083775 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.083739    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:55.126996 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.126975    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"
Sep 09 16:25:55.129040 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.129017    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"
Sep 09 16:25:55.148348 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.148310    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:55.308087 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.308049    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtfpb\" (UniqueName: \"kubernetes.io/projected/2a904a14-212b-42ae-9b1f-ab0ef31a4bac-kube-api-access-vtfpb\") pod \"2a904a14-212b-42ae-9b1f-ab0ef31a4bac\" (UID: \"2a904a14-212b-42ae-9b1f-ab0ef31a4bac\") "
Sep 09 16:25:55.308273 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.308099    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpt7n\" (UniqueName: \"kubernetes.io/projected/c843ebc4-ceb1-43a6-a30a-136f5a93140b-kube-api-access-fpt7n\") pod \"c843ebc4-ceb1-43a6-a30a-136f5a93140b\" (UID: \"c843ebc4-ceb1-43a6-a30a-136f5a93140b\") "
Sep 09 16:25:55.308273 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.308148    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a904a14-212b-42ae-9b1f-ab0ef31a4bac-plugins-config-volume\") pod \"2a904a14-212b-42ae-9b1f-ab0ef31a4bac\" (UID: \"2a904a14-212b-42ae-9b1f-ab0ef31a4bac\") "
Sep 09 16:25:55.308273 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.308199    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c843ebc4-ceb1-43a6-a30a-136f5a93140b-plugins-config-volume\") pod \"c843ebc4-ceb1-43a6-a30a-136f5a93140b\" (UID: \"c843ebc4-ceb1-43a6-a30a-136f5a93140b\") "
Sep 09 16:25:55.308882 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.308544    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a904a14-212b-42ae-9b1f-ab0ef31a4bac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a904a14-212b-42ae-9b1f-ab0ef31a4bac" (UID: "2a904a14-212b-42ae-9b1f-ab0ef31a4bac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:25:55.308882 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.308646    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c843ebc4-ceb1-43a6-a30a-136f5a93140b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c843ebc4-ceb1-43a6-a30a-136f5a93140b" (UID: "c843ebc4-ceb1-43a6-a30a-136f5a93140b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:25:55.310316 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.310297    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c843ebc4-ceb1-43a6-a30a-136f5a93140b-kube-api-access-fpt7n" (OuterVolumeSpecName: "kube-api-access-fpt7n") pod "c843ebc4-ceb1-43a6-a30a-136f5a93140b" (UID: "c843ebc4-ceb1-43a6-a30a-136f5a93140b"). InnerVolumeSpecName "kube-api-access-fpt7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:25:55.310372 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.310352    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a904a14-212b-42ae-9b1f-ab0ef31a4bac-kube-api-access-vtfpb" (OuterVolumeSpecName: "kube-api-access-vtfpb") pod "2a904a14-212b-42ae-9b1f-ab0ef31a4bac" (UID: "2a904a14-212b-42ae-9b1f-ab0ef31a4bac"). InnerVolumeSpecName "kube-api-access-vtfpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:25:55.399668 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.399586    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:25:55.409408 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.409388    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtfpb\" (UniqueName: \"kubernetes.io/projected/2a904a14-212b-42ae-9b1f-ab0ef31a4bac-kube-api-access-vtfpb\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:25:55.409487 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.409410    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpt7n\" (UniqueName: \"kubernetes.io/projected/c843ebc4-ceb1-43a6-a30a-136f5a93140b-kube-api-access-fpt7n\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:25:55.409487 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.409420    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a904a14-212b-42ae-9b1f-ab0ef31a4bac-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:25:55.409487 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.409430    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c843ebc4-ceb1-43a6-a30a-136f5a93140b-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:25:55.491560 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.491523    2556 generic.go:356] "Generic (PLEG): container finished" podID="c843ebc4-ceb1-43a6-a30a-136f5a93140b" containerID="97943b839a74eceb069de6260cf7da32ac9a4014e00d5d2feaf7c50d51eddd71" exitCode=0
Sep 09 16:25:55.491724 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.491573    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"
Sep 09 16:25:55.491724 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.491604    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6" event={"ID":"c843ebc4-ceb1-43a6-a30a-136f5a93140b","Type":"ContainerDied","Data":"97943b839a74eceb069de6260cf7da32ac9a4014e00d5d2feaf7c50d51eddd71"}
Sep 09 16:25:55.491724 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.491662    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6" event={"ID":"c843ebc4-ceb1-43a6-a30a-136f5a93140b","Type":"ContainerDied","Data":"9af44b17412217a04314617772d3f6e416ba6bf8edca9e933651fc61a6aeced0"}
Sep 09 16:25:55.491724 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.491681    2556 scope.go:117] "RemoveContainer" containerID="97943b839a74eceb069de6260cf7da32ac9a4014e00d5d2feaf7c50d51eddd71"
Sep 09 16:25:55.493157 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.493132    2556 generic.go:356] "Generic (PLEG): container finished" podID="2a904a14-212b-42ae-9b1f-ab0ef31a4bac" containerID="4dbff5e7f950993700bbdcffea4ec388b6d76dfccec77f1b3ee476c6194c0432" exitCode=0
Sep 09 16:25:55.493312 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.493159    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn" event={"ID":"2a904a14-212b-42ae-9b1f-ab0ef31a4bac","Type":"ContainerDied","Data":"4dbff5e7f950993700bbdcffea4ec388b6d76dfccec77f1b3ee476c6194c0432"}
Sep 09 16:25:55.493312 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.493203    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"
Sep 09 16:25:55.493312 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.493207    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn" event={"ID":"2a904a14-212b-42ae-9b1f-ab0ef31a4bac","Type":"ContainerDied","Data":"f29872b8942d8eb866f1d55db59748d9dc0362a98b4cd2b76a4f6e1446870a16"}
Sep 09 16:25:55.501123 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.501104    2556 scope.go:117] "RemoveContainer" containerID="97943b839a74eceb069de6260cf7da32ac9a4014e00d5d2feaf7c50d51eddd71"
Sep 09 16:25:55.501454 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:25:55.501425    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97943b839a74eceb069de6260cf7da32ac9a4014e00d5d2feaf7c50d51eddd71\": container with ID starting with 97943b839a74eceb069de6260cf7da32ac9a4014e00d5d2feaf7c50d51eddd71 not found: ID does not exist" containerID="97943b839a74eceb069de6260cf7da32ac9a4014e00d5d2feaf7c50d51eddd71"
Sep 09 16:25:55.501604 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.501488    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97943b839a74eceb069de6260cf7da32ac9a4014e00d5d2feaf7c50d51eddd71"} err="failed to get container status \"97943b839a74eceb069de6260cf7da32ac9a4014e00d5d2feaf7c50d51eddd71\": rpc error: code = NotFound desc = could not find container \"97943b839a74eceb069de6260cf7da32ac9a4014e00d5d2feaf7c50d51eddd71\": container with ID starting with 97943b839a74eceb069de6260cf7da32ac9a4014e00d5d2feaf7c50d51eddd71 not found: ID does not exist"
Sep 09 16:25:55.501604 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.501507    2556 scope.go:117] "RemoveContainer" containerID="4dbff5e7f950993700bbdcffea4ec388b6d76dfccec77f1b3ee476c6194c0432"
Sep 09 16:25:55.511883 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.511856    2556 scope.go:117] "RemoveContainer" containerID="4dbff5e7f950993700bbdcffea4ec388b6d76dfccec77f1b3ee476c6194c0432"
Sep 09 16:25:55.512454 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:25:55.512431    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbff5e7f950993700bbdcffea4ec388b6d76dfccec77f1b3ee476c6194c0432\": container with ID starting with 4dbff5e7f950993700bbdcffea4ec388b6d76dfccec77f1b3ee476c6194c0432 not found: ID does not exist" containerID="4dbff5e7f950993700bbdcffea4ec388b6d76dfccec77f1b3ee476c6194c0432"
Sep 09 16:25:55.512588 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.512459    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbff5e7f950993700bbdcffea4ec388b6d76dfccec77f1b3ee476c6194c0432"} err="failed to get container status \"4dbff5e7f950993700bbdcffea4ec388b6d76dfccec77f1b3ee476c6194c0432\": rpc error: code = NotFound desc = could not find container \"4dbff5e7f950993700bbdcffea4ec388b6d76dfccec77f1b3ee476c6194c0432\": container with ID starting with 4dbff5e7f950993700bbdcffea4ec388b6d76dfccec77f1b3ee476c6194c0432 not found: ID does not exist"
Sep 09 16:25:55.517196 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.517175    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"]
Sep 09 16:25:55.520868 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.520848    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-131689-66476565d7-pvwm6"]
Sep 09 16:25:55.530663 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.530641    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"]
Sep 09 16:25:55.533883 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.533865    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-131689-56d894f5f4-8jjfn"]
Sep 09 16:25:55.763297 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.763268    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a904a14-212b-42ae-9b1f-ab0ef31a4bac" path="/var/lib/kubelet/pods/2a904a14-212b-42ae-9b1f-ab0ef31a4bac/volumes"
Sep 09 16:25:55.763648 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.763635    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c843ebc4-ceb1-43a6-a30a-136f5a93140b" path="/var/lib/kubelet/pods/c843ebc4-ceb1-43a6-a30a-136f5a93140b/volumes"
Sep 09 16:25:55.763921 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:25:55.763910    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f336017a-23c7-4a3e-b94b-37415d5bd4a7" path="/var/lib/kubelet/pods/f336017a-23c7-4a3e-b94b-37415d5bd4a7/volumes"
Sep 09 16:26:01.103441 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:01.103345    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:26:01.203880 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:01.203815    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:26:01.352703 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:01.352656    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:26:01.793964 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:01.793934    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:26:12.153374 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.153342    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:26:12.178450 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.178416    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"]
Sep 09 16:26:12.178719 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.178678    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r" podUID="35fc4078-ba08-4a23-a3f8-52463674c33d" containerName="payload-processing" containerID="cri-o://77652ad59dcf2cbcf2a7efd337353a17c260ce47b13170ff9631111b977e593a" gracePeriod=30
Sep 09 16:26:12.183002 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.182969    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"]
Sep 09 16:26:12.184300 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.184248    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw" podUID="1b089384-dc8c-440c-aa46-dcce162ef93c" containerName="payload-pre-processing" containerID="cri-o://63bd422f5f68a6330f269302081ab60791d151c948efa0a594f335a1b727a611" gracePeriod=30
Sep 09 16:26:12.215049 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.215018    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:26:12.315094 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.315053    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:26:12.368741 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.368713    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:26:12.448756 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.448731    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"
Sep 09 16:26:12.451538 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.451518    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"
Sep 09 16:26:12.544052 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.544018    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2j2b\" (UniqueName: \"kubernetes.io/projected/35fc4078-ba08-4a23-a3f8-52463674c33d-kube-api-access-x2j2b\") pod \"35fc4078-ba08-4a23-a3f8-52463674c33d\" (UID: \"35fc4078-ba08-4a23-a3f8-52463674c33d\") "
Sep 09 16:26:12.544052 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.544062    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b089384-dc8c-440c-aa46-dcce162ef93c-plugins-config-volume\") pod \"1b089384-dc8c-440c-aa46-dcce162ef93c\" (UID: \"1b089384-dc8c-440c-aa46-dcce162ef93c\") "
Sep 09 16:26:12.544321 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.544105    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bprmg\" (UniqueName: \"kubernetes.io/projected/1b089384-dc8c-440c-aa46-dcce162ef93c-kube-api-access-bprmg\") pod \"1b089384-dc8c-440c-aa46-dcce162ef93c\" (UID: \"1b089384-dc8c-440c-aa46-dcce162ef93c\") "
Sep 09 16:26:12.544321 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.544181    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35fc4078-ba08-4a23-a3f8-52463674c33d-plugins-config-volume\") pod \"35fc4078-ba08-4a23-a3f8-52463674c33d\" (UID: \"35fc4078-ba08-4a23-a3f8-52463674c33d\") "
Sep 09 16:26:12.544650 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.544615    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b089384-dc8c-440c-aa46-dcce162ef93c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1b089384-dc8c-440c-aa46-dcce162ef93c" (UID: "1b089384-dc8c-440c-aa46-dcce162ef93c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:26:12.544650 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.544624    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fc4078-ba08-4a23-a3f8-52463674c33d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "35fc4078-ba08-4a23-a3f8-52463674c33d" (UID: "35fc4078-ba08-4a23-a3f8-52463674c33d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:26:12.546218 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.546196    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b089384-dc8c-440c-aa46-dcce162ef93c-kube-api-access-bprmg" (OuterVolumeSpecName: "kube-api-access-bprmg") pod "1b089384-dc8c-440c-aa46-dcce162ef93c" (UID: "1b089384-dc8c-440c-aa46-dcce162ef93c"). InnerVolumeSpecName "kube-api-access-bprmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:26:12.546321 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.546226    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fc4078-ba08-4a23-a3f8-52463674c33d-kube-api-access-x2j2b" (OuterVolumeSpecName: "kube-api-access-x2j2b") pod "35fc4078-ba08-4a23-a3f8-52463674c33d" (UID: "35fc4078-ba08-4a23-a3f8-52463674c33d"). InnerVolumeSpecName "kube-api-access-x2j2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:26:12.580380 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.580347    2556 generic.go:356] "Generic (PLEG): container finished" podID="1b089384-dc8c-440c-aa46-dcce162ef93c" containerID="63bd422f5f68a6330f269302081ab60791d151c948efa0a594f335a1b727a611" exitCode=0
Sep 09 16:26:12.580540 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.580395    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"
Sep 09 16:26:12.580540 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.580426    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw" event={"ID":"1b089384-dc8c-440c-aa46-dcce162ef93c","Type":"ContainerDied","Data":"63bd422f5f68a6330f269302081ab60791d151c948efa0a594f335a1b727a611"}
Sep 09 16:26:12.580540 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.580509    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw" event={"ID":"1b089384-dc8c-440c-aa46-dcce162ef93c","Type":"ContainerDied","Data":"87d33c16fa56adfc88d6008fb50dfdc932f8623120e2d85a3f0b4643c68ec997"}
Sep 09 16:26:12.580540 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.580536    2556 scope.go:117] "RemoveContainer" containerID="63bd422f5f68a6330f269302081ab60791d151c948efa0a594f335a1b727a611"
Sep 09 16:26:12.581883 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.581848    2556 generic.go:356] "Generic (PLEG): container finished" podID="35fc4078-ba08-4a23-a3f8-52463674c33d" containerID="77652ad59dcf2cbcf2a7efd337353a17c260ce47b13170ff9631111b977e593a" exitCode=0
Sep 09 16:26:12.581883 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.581880    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r" event={"ID":"35fc4078-ba08-4a23-a3f8-52463674c33d","Type":"ContainerDied","Data":"77652ad59dcf2cbcf2a7efd337353a17c260ce47b13170ff9631111b977e593a"}
Sep 09 16:26:12.582041 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.581902    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"
Sep 09 16:26:12.582041 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.581917    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r" event={"ID":"35fc4078-ba08-4a23-a3f8-52463674c33d","Type":"ContainerDied","Data":"c25591d1901ecf8d87d1137791304ac6ea8729c56610a538ef9429ece14375a2"}
Sep 09 16:26:12.589620 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.589597    2556 scope.go:117] "RemoveContainer" containerID="63bd422f5f68a6330f269302081ab60791d151c948efa0a594f335a1b727a611"
Sep 09 16:26:12.590074 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:26:12.590044    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63bd422f5f68a6330f269302081ab60791d151c948efa0a594f335a1b727a611\": container with ID starting with 63bd422f5f68a6330f269302081ab60791d151c948efa0a594f335a1b727a611 not found: ID does not exist" containerID="63bd422f5f68a6330f269302081ab60791d151c948efa0a594f335a1b727a611"
Sep 09 16:26:12.590183 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.590083    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63bd422f5f68a6330f269302081ab60791d151c948efa0a594f335a1b727a611"} err="failed to get container status \"63bd422f5f68a6330f269302081ab60791d151c948efa0a594f335a1b727a611\": rpc error: code = NotFound desc = could not find container \"63bd422f5f68a6330f269302081ab60791d151c948efa0a594f335a1b727a611\": container with ID starting with 63bd422f5f68a6330f269302081ab60791d151c948efa0a594f335a1b727a611 not found: ID does not exist"
Sep 09 16:26:12.590183 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.590106    2556 scope.go:117] "RemoveContainer" containerID="77652ad59dcf2cbcf2a7efd337353a17c260ce47b13170ff9631111b977e593a"
Sep 09 16:26:12.597944 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.597926    2556 scope.go:117] "RemoveContainer" containerID="77652ad59dcf2cbcf2a7efd337353a17c260ce47b13170ff9631111b977e593a"
Sep 09 16:26:12.598199 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:26:12.598179    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77652ad59dcf2cbcf2a7efd337353a17c260ce47b13170ff9631111b977e593a\": container with ID starting with 77652ad59dcf2cbcf2a7efd337353a17c260ce47b13170ff9631111b977e593a not found: ID does not exist" containerID="77652ad59dcf2cbcf2a7efd337353a17c260ce47b13170ff9631111b977e593a"
Sep 09 16:26:12.598282 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.598205    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77652ad59dcf2cbcf2a7efd337353a17c260ce47b13170ff9631111b977e593a"} err="failed to get container status \"77652ad59dcf2cbcf2a7efd337353a17c260ce47b13170ff9631111b977e593a\": rpc error: code = NotFound desc = could not find container \"77652ad59dcf2cbcf2a7efd337353a17c260ce47b13170ff9631111b977e593a\": container with ID starting with 77652ad59dcf2cbcf2a7efd337353a17c260ce47b13170ff9631111b977e593a not found: ID does not exist"
Sep 09 16:26:12.604196 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.604174    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"]
Sep 09 16:26:12.608406 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.608386    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-13d59f-8489f9dcf8-9xjcw"]
Sep 09 16:26:12.618086 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.618064    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"]
Sep 09 16:26:12.621145 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.621125    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-13d59f-577b867d9-pm96r"]
Sep 09 16:26:12.645557 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.645517    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2j2b\" (UniqueName: \"kubernetes.io/projected/35fc4078-ba08-4a23-a3f8-52463674c33d-kube-api-access-x2j2b\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:26:12.645557 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.645554    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b089384-dc8c-440c-aa46-dcce162ef93c-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:26:12.645557 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.645565    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bprmg\" (UniqueName: \"kubernetes.io/projected/1b089384-dc8c-440c-aa46-dcce162ef93c-kube-api-access-bprmg\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:26:12.645557 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.645574    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35fc4078-ba08-4a23-a3f8-52463674c33d-plugins-config-volume\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:26:12.719525 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:12.719409    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:26:13.762849 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:13.762817    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b089384-dc8c-440c-aa46-dcce162ef93c" path="/var/lib/kubelet/pods/1b089384-dc8c-440c-aa46-dcce162ef93c/volumes"
Sep 09 16:26:13.763240 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:13.763159    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35fc4078-ba08-4a23-a3f8-52463674c33d" path="/var/lib/kubelet/pods/35fc4078-ba08-4a23-a3f8-52463674c33d/volumes"
Sep 09 16:26:15.259638 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:15.259594    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:26:18.520398 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:18.520341    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:26:18.587173 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:18.587139    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:26:18.782963 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:18.782877    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:26:35.745328 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:35.745302    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 16:26:35.749766 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:26:35.749744    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 16:27:44.962004 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:27:44.961909    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:27:47.795274 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:27:47.795210    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:28:17.183566 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:28:17.183524    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:28:17.254071 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:28:17.254041    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:29:48.777151 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:29:48.777106    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:30:00.144219 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144180    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816190-szcbb"]
Sep 09 16:30:00.144647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144498    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c843ebc4-ceb1-43a6-a30a-136f5a93140b" containerName="payload-processing"
Sep 09 16:30:00.144647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144509    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c843ebc4-ceb1-43a6-a30a-136f5a93140b" containerName="payload-processing"
Sep 09 16:30:00.144647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144525    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35fc4078-ba08-4a23-a3f8-52463674c33d" containerName="payload-processing"
Sep 09 16:30:00.144647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144533    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="35fc4078-ba08-4a23-a3f8-52463674c33d" containerName="payload-processing"
Sep 09 16:30:00.144647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144539    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f336017a-23c7-4a3e-b94b-37415d5bd4a7" containerName="revoke-keys"
Sep 09 16:30:00.144647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144544    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f336017a-23c7-4a3e-b94b-37415d5bd4a7" containerName="revoke-keys"
Sep 09 16:30:00.144647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144552    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a904a14-212b-42ae-9b1f-ab0ef31a4bac" containerName="payload-pre-processing"
Sep 09 16:30:00.144647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144557    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a904a14-212b-42ae-9b1f-ab0ef31a4bac" containerName="payload-pre-processing"
Sep 09 16:30:00.144647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144568    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b089384-dc8c-440c-aa46-dcce162ef93c" containerName="payload-pre-processing"
Sep 09 16:30:00.144647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144574    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b089384-dc8c-440c-aa46-dcce162ef93c" containerName="payload-pre-processing"
Sep 09 16:30:00.144647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144625    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="35fc4078-ba08-4a23-a3f8-52463674c33d" containerName="payload-processing"
Sep 09 16:30:00.144647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144634    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a904a14-212b-42ae-9b1f-ab0ef31a4bac" containerName="payload-pre-processing"
Sep 09 16:30:00.144647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144640    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c843ebc4-ceb1-43a6-a30a-136f5a93140b" containerName="payload-processing"
Sep 09 16:30:00.144647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144645    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f336017a-23c7-4a3e-b94b-37415d5bd4a7" containerName="revoke-keys"
Sep 09 16:30:00.144647 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.144651    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b089384-dc8c-440c-aa46-dcce162ef93c" containerName="payload-pre-processing"
Sep 09 16:30:00.147431 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.147412    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816190-szcbb"
Sep 09 16:30:00.149646 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.149621    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\""
Sep 09 16:30:00.154032 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.154002    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816190-szcbb"]
Sep 09 16:30:00.161878 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.161848    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816190-szcbb"
Sep 09 16:30:00.430099 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.430069    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816190-szcbb"]
Sep 09 16:30:00.432559 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:30:00.432530    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc278d688_07e3_45a3_9537_62fa638d88b8.slice/crio-b11d7da846ebbe7cd4544e5cc40ab91534dd6c642702bae6bff378134ffcc0ed WatchSource:0}: Error finding container b11d7da846ebbe7cd4544e5cc40ab91534dd6c642702bae6bff378134ffcc0ed: Status 404 returned error can't find the container with id b11d7da846ebbe7cd4544e5cc40ab91534dd6c642702bae6bff378134ffcc0ed
Sep 09 16:30:00.576052 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:00.576019    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816190-szcbb" event={"ID":"c278d688-07e3-45a3-9537-62fa638d88b8","Type":"ContainerStarted","Data":"b11d7da846ebbe7cd4544e5cc40ab91534dd6c642702bae6bff378134ffcc0ed"}
Sep 09 16:30:01.582192 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:01.582089    2556 generic.go:356] "Generic (PLEG): container finished" podID="c278d688-07e3-45a3-9537-62fa638d88b8" containerID="5acb4f37d61e249957f616222bf47bdf02c5114001a167f7f796b8887b9810e3" exitCode=0
Sep 09 16:30:01.582192 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:01.582139    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816190-szcbb" event={"ID":"c278d688-07e3-45a3-9537-62fa638d88b8","Type":"ContainerDied","Data":"5acb4f37d61e249957f616222bf47bdf02c5114001a167f7f796b8887b9810e3"}
Sep 09 16:30:02.716040 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:02.716011    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816190-szcbb"
Sep 09 16:30:03.591695 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:03.591655    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816190-szcbb"
Sep 09 16:30:03.591695 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:03.591668    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816190-szcbb" event={"ID":"c278d688-07e3-45a3-9537-62fa638d88b8","Type":"ContainerDied","Data":"b11d7da846ebbe7cd4544e5cc40ab91534dd6c642702bae6bff378134ffcc0ed"}
Sep 09 16:30:03.591919 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:03.591723    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b11d7da846ebbe7cd4544e5cc40ab91534dd6c642702bae6bff378134ffcc0ed"
Sep 09 16:30:36.008557 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:36.008477    2556 scope.go:117] "RemoveContainer" containerID="975e8b5e2025f657213203b210f0fb45230f40664db60db412b596cedfed9500"
Sep 09 16:30:36.016841 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:36.016819    2556 scope.go:117] "RemoveContainer" containerID="3b0495de82a6d0a77e4b105fc52aebc57421da1beea35d90269317c5461e4d1b"
Sep 09 16:30:55.038055 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.038017    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"]
Sep 09 16:30:55.038526 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.038477    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c278d688-07e3-45a3-9537-62fa638d88b8" containerName="cleanup"
Sep 09 16:30:55.038526 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.038496    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c278d688-07e3-45a3-9537-62fa638d88b8" containerName="cleanup"
Sep 09 16:30:55.038607 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.038592    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c278d688-07e3-45a3-9537-62fa638d88b8" containerName="cleanup"
Sep 09 16:30:55.041781 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.041765    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.043903 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.043880    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-fc21836e-kserve-self-signed-certs\""
Sep 09 16:30:55.050772 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.050745    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"]
Sep 09 16:30:55.079993 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.079968    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:30:55.164294 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.164254    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca69816-74f7-4bf0-b800-fe10aadfde74-tls-certs\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.164294 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.164303    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-tmp-dir\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.164561 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.164340    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-model-cache\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.164561 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.164362    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-dshm\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.164561 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.164384    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgkv8\" (UniqueName: \"kubernetes.io/projected/6ca69816-74f7-4bf0-b800-fe10aadfde74-kube-api-access-dgkv8\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.164561 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.164473    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-home\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.265660 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.265619    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-model-cache\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.265822 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.265682    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-dshm\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.265822 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.265709    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgkv8\" (UniqueName: \"kubernetes.io/projected/6ca69816-74f7-4bf0-b800-fe10aadfde74-kube-api-access-dgkv8\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.265822 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.265754    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-home\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.265822 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.265778    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca69816-74f7-4bf0-b800-fe10aadfde74-tls-certs\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.265822 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.265811    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-tmp-dir\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.266072 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.265966    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-model-cache\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.266072 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.266060    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-home\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.266187 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.266119    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-tmp-dir\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.267952 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.267931    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-dshm\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.268230 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.268209    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca69816-74f7-4bf0-b800-fe10aadfde74-tls-certs\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.273826 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.273803    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgkv8\" (UniqueName: \"kubernetes.io/projected/6ca69816-74f7-4bf0-b800-fe10aadfde74-kube-api-access-dgkv8\") pod \"e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") " pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.351590 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.351497    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:30:55.622071 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.622047    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"]
Sep 09 16:30:55.624224 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:30:55.624195    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ca69816_74f7_4bf0_b800_fe10aadfde74.slice/crio-dffd1a98b9da49007c7b0a67f9800d93d771a1f9a68fd296ce4c4224489048f9 WatchSource:0}: Error finding container dffd1a98b9da49007c7b0a67f9800d93d771a1f9a68fd296ce4c4224489048f9: Status 404 returned error can't find the container with id dffd1a98b9da49007c7b0a67f9800d93d771a1f9a68fd296ce4c4224489048f9
Sep 09 16:30:55.823809 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.823773    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk" event={"ID":"6ca69816-74f7-4bf0-b800-fe10aadfde74","Type":"ContainerStarted","Data":"73ccfa1db62c043ae1693f03ef46c6f977504a2e1df1c7e674f23e549d72a6f6"}
Sep 09 16:30:55.823809 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.823811    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk" event={"ID":"6ca69816-74f7-4bf0-b800-fe10aadfde74","Type":"ContainerStarted","Data":"dffd1a98b9da49007c7b0a67f9800d93d771a1f9a68fd296ce4c4224489048f9"}
Sep 09 16:30:55.839289 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:30:55.839239    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk" podStartSLOduration=0.839223146 podStartE2EDuration="839.223146ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:30:55.839064279 +0000 UTC m=+2360.705575522" watchObservedRunningTime="2026-09-09 16:30:55.839223146 +0000 UTC m=+2360.705734367"
Sep 09 16:31:02.619214 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:02.619177    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:31:05.352348 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:05.352304    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:31:05.352348 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:05.352357    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:31:05.364877 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:05.364848    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:31:05.877785 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:05.877754    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:31:17.043333 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.043301    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"]
Sep 09 16:31:17.046316 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.046297    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.048637 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.048612    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-fc21836e-kserve-self-signed-certs\""
Sep 09 16:31:17.057886 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.057864    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"]
Sep 09 16:31:17.075234 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.075194    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:31:17.143531 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.143501    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-dshm\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.143531 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.143533    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-tmp-dir\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.143737 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.143569    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-model-cache\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.143737 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.143649    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d10e5f-5e4e-411c-b162-7653c4837b0a-tls-certs\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.143737 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.143715    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-home\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.143899 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.143745    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn242\" (UniqueName: \"kubernetes.io/projected/92d10e5f-5e4e-411c-b162-7653c4837b0a-kube-api-access-zn242\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.244858 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.244820    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d10e5f-5e4e-411c-b162-7653c4837b0a-tls-certs\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.245050 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.244882    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-home\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.245050 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.244908    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn242\" (UniqueName: \"kubernetes.io/projected/92d10e5f-5e4e-411c-b162-7653c4837b0a-kube-api-access-zn242\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.245050 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.244927    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-dshm\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.245050 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.244942    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-tmp-dir\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.245050 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.244980    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-model-cache\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.245441 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.245418    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-model-cache\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.245441 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.245432    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-home\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.245613 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.245509    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-tmp-dir\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.247288 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.247267    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-dshm\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.247552 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.247532    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d10e5f-5e4e-411c-b162-7653c4837b0a-tls-certs\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.251834 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.251818    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn242\" (UniqueName: \"kubernetes.io/projected/92d10e5f-5e4e-411c-b162-7653c4837b0a-kube-api-access-zn242\") pod \"e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") " pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.355771 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.355694    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:17.624964 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.624878    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"]
Sep 09 16:31:17.627414 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:31:17.627387    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92d10e5f_5e4e_411c_b162_7653c4837b0a.slice/crio-a541f0d59aa94bdc66f760515efda6a0464a6e442357a94c213059c0fcf54b89 WatchSource:0}: Error finding container a541f0d59aa94bdc66f760515efda6a0464a6e442357a94c213059c0fcf54b89: Status 404 returned error can't find the container with id a541f0d59aa94bdc66f760515efda6a0464a6e442357a94c213059c0fcf54b89
Sep 09 16:31:17.917778 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.917689    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl" event={"ID":"92d10e5f-5e4e-411c-b162-7653c4837b0a","Type":"ContainerStarted","Data":"9f57675a22e679efa7b03168801312b324286a86f9986081bb75e1caa8048511"}
Sep 09 16:31:17.917778 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.917729    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl" event={"ID":"92d10e5f-5e4e-411c-b162-7653c4837b0a","Type":"ContainerStarted","Data":"a541f0d59aa94bdc66f760515efda6a0464a6e442357a94c213059c0fcf54b89"}
Sep 09 16:31:17.931816 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:17.931769    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl" podStartSLOduration=0.931753395 podStartE2EDuration="931.753395ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:31:17.931165568 +0000 UTC m=+2382.797676782" watchObservedRunningTime="2026-09-09 16:31:17.931753395 +0000 UTC m=+2382.798264614"
Sep 09 16:31:27.356672 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:27.356637    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:27.357191 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:27.356702    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:27.369161 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:27.369136    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:27.970904 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:27.970872    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:35.582664 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:35.582632    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"]
Sep 09 16:31:35.601456 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:35.601422    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:31:35.771187 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:35.771161    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 16:31:35.777683 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:35.777660    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 16:31:36.039921 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:36.039892    2556 scope.go:117] "RemoveContainer" containerID="bb4fd02fc3ff718c4329e98f4ccb4f9f36708f82f92a5f4c6e06f7eda016a92a"
Sep 09 16:31:36.823126 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:36.823091    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"]
Sep 09 16:31:36.847959 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:36.847932    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:31:50.602261 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:50.602191    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk" podUID="6ca69816-74f7-4bf0-b800-fe10aadfde74" containerName="main" containerID="cri-o://73ccfa1db62c043ae1693f03ef46c6f977504a2e1df1c7e674f23e549d72a6f6" gracePeriod=45
Sep 09 16:31:50.838684 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:50.838660    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:31:51.027581 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.027554    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-model-cache\") pod \"6ca69816-74f7-4bf0-b800-fe10aadfde74\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") "
Sep 09 16:31:51.027788 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.027589    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-tmp-dir\") pod \"6ca69816-74f7-4bf0-b800-fe10aadfde74\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") "
Sep 09 16:31:51.027788 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.027613    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-dshm\") pod \"6ca69816-74f7-4bf0-b800-fe10aadfde74\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") "
Sep 09 16:31:51.027788 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.027681    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgkv8\" (UniqueName: \"kubernetes.io/projected/6ca69816-74f7-4bf0-b800-fe10aadfde74-kube-api-access-dgkv8\") pod \"6ca69816-74f7-4bf0-b800-fe10aadfde74\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") "
Sep 09 16:31:51.027788 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.027713    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-home\") pod \"6ca69816-74f7-4bf0-b800-fe10aadfde74\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") "
Sep 09 16:31:51.027788 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.027733    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca69816-74f7-4bf0-b800-fe10aadfde74-tls-certs\") pod \"6ca69816-74f7-4bf0-b800-fe10aadfde74\" (UID: \"6ca69816-74f7-4bf0-b800-fe10aadfde74\") "
Sep 09 16:31:51.028034 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.027858    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ca69816-74f7-4bf0-b800-fe10aadfde74" (UID: "6ca69816-74f7-4bf0-b800-fe10aadfde74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:31:51.028034 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.027874    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ca69816-74f7-4bf0-b800-fe10aadfde74" (UID: "6ca69816-74f7-4bf0-b800-fe10aadfde74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:31:51.028034 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.027989    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-home" (OuterVolumeSpecName: "home") pod "6ca69816-74f7-4bf0-b800-fe10aadfde74" (UID: "6ca69816-74f7-4bf0-b800-fe10aadfde74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:31:51.029931 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.029900    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca69816-74f7-4bf0-b800-fe10aadfde74-kube-api-access-dgkv8" (OuterVolumeSpecName: "kube-api-access-dgkv8") pod "6ca69816-74f7-4bf0-b800-fe10aadfde74" (UID: "6ca69816-74f7-4bf0-b800-fe10aadfde74"). InnerVolumeSpecName "kube-api-access-dgkv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:31:51.030073 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.029956    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-dshm" (OuterVolumeSpecName: "dshm") pod "6ca69816-74f7-4bf0-b800-fe10aadfde74" (UID: "6ca69816-74f7-4bf0-b800-fe10aadfde74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:31:51.030073 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.030030    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ca69816-74f7-4bf0-b800-fe10aadfde74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ca69816-74f7-4bf0-b800-fe10aadfde74" (UID: "6ca69816-74f7-4bf0-b800-fe10aadfde74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:31:51.062320 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.062292    2556 generic.go:356] "Generic (PLEG): container finished" podID="6ca69816-74f7-4bf0-b800-fe10aadfde74" containerID="73ccfa1db62c043ae1693f03ef46c6f977504a2e1df1c7e674f23e549d72a6f6" exitCode=0
Sep 09 16:31:51.062433 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.062335    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"
Sep 09 16:31:51.062433 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.062375    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk" event={"ID":"6ca69816-74f7-4bf0-b800-fe10aadfde74","Type":"ContainerDied","Data":"73ccfa1db62c043ae1693f03ef46c6f977504a2e1df1c7e674f23e549d72a6f6"}
Sep 09 16:31:51.062433 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.062417    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk" event={"ID":"6ca69816-74f7-4bf0-b800-fe10aadfde74","Type":"ContainerDied","Data":"dffd1a98b9da49007c7b0a67f9800d93d771a1f9a68fd296ce4c4224489048f9"}
Sep 09 16:31:51.062587 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.062435    2556 scope.go:117] "RemoveContainer" containerID="73ccfa1db62c043ae1693f03ef46c6f977504a2e1df1c7e674f23e549d72a6f6"
Sep 09 16:31:51.071321 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.071302    2556 scope.go:117] "RemoveContainer" containerID="73ccfa1db62c043ae1693f03ef46c6f977504a2e1df1c7e674f23e549d72a6f6"
Sep 09 16:31:51.071598 ip-10-0-133-93 kubenswrapper[2556]: E0909 16:31:51.071579    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ccfa1db62c043ae1693f03ef46c6f977504a2e1df1c7e674f23e549d72a6f6\": container with ID starting with 73ccfa1db62c043ae1693f03ef46c6f977504a2e1df1c7e674f23e549d72a6f6 not found: ID does not exist" containerID="73ccfa1db62c043ae1693f03ef46c6f977504a2e1df1c7e674f23e549d72a6f6"
Sep 09 16:31:51.071646 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.071606    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ccfa1db62c043ae1693f03ef46c6f977504a2e1df1c7e674f23e549d72a6f6"} err="failed to get container status \"73ccfa1db62c043ae1693f03ef46c6f977504a2e1df1c7e674f23e549d72a6f6\": rpc error: code = NotFound desc = could not find container \"73ccfa1db62c043ae1693f03ef46c6f977504a2e1df1c7e674f23e549d72a6f6\": container with ID starting with 73ccfa1db62c043ae1693f03ef46c6f977504a2e1df1c7e674f23e549d72a6f6 not found: ID does not exist"
Sep 09 16:31:51.083244 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.083220    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"]
Sep 09 16:31:51.086981 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.086959    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-fc21836e-kserve-d565675c5-ljqmk"]
Sep 09 16:31:51.128694 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.128674    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgkv8\" (UniqueName: \"kubernetes.io/projected/6ca69816-74f7-4bf0-b800-fe10aadfde74-kube-api-access-dgkv8\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:31:51.128694 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.128694    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-home\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:31:51.128844 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.128704    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca69816-74f7-4bf0-b800-fe10aadfde74-tls-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:31:51.128844 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.128713    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-model-cache\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:31:51.128844 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.128722    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-tmp-dir\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:31:51.128844 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.128729    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca69816-74f7-4bf0-b800-fe10aadfde74-dshm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:31:51.763137 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.763095    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ca69816-74f7-4bf0-b800-fe10aadfde74" path="/var/lib/kubelet/pods/6ca69816-74f7-4bf0-b800-fe10aadfde74/volumes"
Sep 09 16:31:51.843352 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:51.843309    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl" podUID="92d10e5f-5e4e-411c-b162-7653c4837b0a" containerName="main" containerID="cri-o://9f57675a22e679efa7b03168801312b324286a86f9986081bb75e1caa8048511" gracePeriod=45
Sep 09 16:31:52.070361 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.070271    2556 generic.go:356] "Generic (PLEG): container finished" podID="92d10e5f-5e4e-411c-b162-7653c4837b0a" containerID="9f57675a22e679efa7b03168801312b324286a86f9986081bb75e1caa8048511" exitCode=0
Sep 09 16:31:52.070361 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.070341    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl" event={"ID":"92d10e5f-5e4e-411c-b162-7653c4837b0a","Type":"ContainerDied","Data":"9f57675a22e679efa7b03168801312b324286a86f9986081bb75e1caa8048511"}
Sep 09 16:31:52.090113 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.090089    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:52.238653 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.238618    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d10e5f-5e4e-411c-b162-7653c4837b0a-tls-certs\") pod \"92d10e5f-5e4e-411c-b162-7653c4837b0a\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") "
Sep 09 16:31:52.238653 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.238665    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-tmp-dir\") pod \"92d10e5f-5e4e-411c-b162-7653c4837b0a\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") "
Sep 09 16:31:52.238903 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.238695    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-model-cache\") pod \"92d10e5f-5e4e-411c-b162-7653c4837b0a\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") "
Sep 09 16:31:52.238903 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.238722    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn242\" (UniqueName: \"kubernetes.io/projected/92d10e5f-5e4e-411c-b162-7653c4837b0a-kube-api-access-zn242\") pod \"92d10e5f-5e4e-411c-b162-7653c4837b0a\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") "
Sep 09 16:31:52.238903 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.238747    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-home\") pod \"92d10e5f-5e4e-411c-b162-7653c4837b0a\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") "
Sep 09 16:31:52.238903 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.238804    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-dshm\") pod \"92d10e5f-5e4e-411c-b162-7653c4837b0a\" (UID: \"92d10e5f-5e4e-411c-b162-7653c4837b0a\") "
Sep 09 16:31:52.239140 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.238946    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92d10e5f-5e4e-411c-b162-7653c4837b0a" (UID: "92d10e5f-5e4e-411c-b162-7653c4837b0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:31:52.239140 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.239026    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-model-cache" (OuterVolumeSpecName: "model-cache") pod "92d10e5f-5e4e-411c-b162-7653c4837b0a" (UID: "92d10e5f-5e4e-411c-b162-7653c4837b0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:31:52.239140 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.239052    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-home" (OuterVolumeSpecName: "home") pod "92d10e5f-5e4e-411c-b162-7653c4837b0a" (UID: "92d10e5f-5e4e-411c-b162-7653c4837b0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:31:52.240799 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.240775    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-dshm" (OuterVolumeSpecName: "dshm") pod "92d10e5f-5e4e-411c-b162-7653c4837b0a" (UID: "92d10e5f-5e4e-411c-b162-7653c4837b0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:31:52.240910 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.240874    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92d10e5f-5e4e-411c-b162-7653c4837b0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92d10e5f-5e4e-411c-b162-7653c4837b0a" (UID: "92d10e5f-5e4e-411c-b162-7653c4837b0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:31:52.240956 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.240916    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d10e5f-5e4e-411c-b162-7653c4837b0a-kube-api-access-zn242" (OuterVolumeSpecName: "kube-api-access-zn242") pod "92d10e5f-5e4e-411c-b162-7653c4837b0a" (UID: "92d10e5f-5e4e-411c-b162-7653c4837b0a"). InnerVolumeSpecName "kube-api-access-zn242". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:31:52.340046 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.339962    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d10e5f-5e4e-411c-b162-7653c4837b0a-tls-certs\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:31:52.340046 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.339999    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-tmp-dir\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:31:52.340046 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.340013    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-model-cache\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:31:52.340046 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.340025    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn242\" (UniqueName: \"kubernetes.io/projected/92d10e5f-5e4e-411c-b162-7653c4837b0a-kube-api-access-zn242\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:31:52.340046 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.340038    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-home\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:31:52.340046 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:52.340052    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d10e5f-5e4e-411c-b162-7653c4837b0a-dshm\") on node \"ip-10-0-133-93.ec2.internal\" DevicePath \"\""
Sep 09 16:31:53.076786 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:53.076757    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"
Sep 09 16:31:53.077236 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:53.076759    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl" event={"ID":"92d10e5f-5e4e-411c-b162-7653c4837b0a","Type":"ContainerDied","Data":"a541f0d59aa94bdc66f760515efda6a0464a6e442357a94c213059c0fcf54b89"}
Sep 09 16:31:53.077236 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:53.076884    2556 scope.go:117] "RemoveContainer" containerID="9f57675a22e679efa7b03168801312b324286a86f9986081bb75e1caa8048511"
Sep 09 16:31:53.097130 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:53.097105    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"]
Sep 09 16:31:53.100639 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:53.100615    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-fc21836e-kserve-556f9bb487-744fl"]
Sep 09 16:31:53.762775 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:31:53.762742    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92d10e5f-5e4e-411c-b162-7653c4837b0a" path="/var/lib/kubelet/pods/92d10e5f-5e4e-411c-b162-7653c4837b0a/volumes"
Sep 09 16:32:12.857259 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:12.857222    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:32:12.958016 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:12.957983    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:32:27.350074 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:27.350034    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6574f9ffbb-tt6rp_56f9152d-41ed-4827-8f66-c3b2b9af5c15/authorino/0.log"
Sep 09 16:32:33.834693 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:33.834661    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4ccd886c-lfmxt_efca11da-bf8c-469c-b9e1-904559b5c395/copy-manifests/0.log"
Sep 09 16:32:33.841481 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:33.841440    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4ccd886c-lfmxt_efca11da-bf8c-469c-b9e1-904559b5c395/manager/0.log"
Sep 09 16:32:34.570103 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:34.570078    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816175-trxmj_6aa9cdce-7e7f-4c88-a75c-7aaf5d98c5f7/cleanup/0.log"
Sep 09 16:32:34.693860 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:34.693828    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816190-szcbb_c278d688-07e3-45a3-9537-62fa638d88b8/cleanup/0.log"
Sep 09 16:32:35.624606 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:35.624573    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6574f9ffbb-tt6rp_56f9152d-41ed-4827-8f66-c3b2b9af5c15/authorino/0.log"
Sep 09 16:32:35.976724 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:35.976696    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9vfgr_bc269cb7-8c85-49e5-96cd-aac33a0f7cf8/kuadrant-console-plugin/0.log"
Sep 09 16:32:36.057316 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:36.057283    2556 scope.go:117] "RemoveContainer" containerID="c706840b43a97e148b24e7ce51d1f96ca52240bacfb35fc4efc316511e520ac7"
Sep 09 16:32:37.075886 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:37.075857    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-747b697884-whtvt_4972d020-567d-464e-8f08-21ca7620cab0/kube-auth-proxy/0.log"
Sep 09 16:32:37.346010 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:37.345930    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-zw24c_d6653c3a-2dbc-41af-8b21-9ca857ba26f4/payload-pre-processing/0.log"
Sep 09 16:32:37.381091 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:37.381058    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:32:37.964594 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:37.964560    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-kkcth_8fba577e-4974-4fc1-b73b-65ff011560b5/main/0.log"
Sep 09 16:32:38.092421 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:38.092395    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-pppdx_c3cd58b7-297c-4b14-b353-e029001f2797/main/0.log"
Sep 09 16:32:38.336565 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:38.336484    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-48ghc_e9de4b71-89ec-4a22-8402-3054f4411a5c/main/0.log"
Sep 09 16:32:44.093075 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.093040    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g6q5c/must-gather-6xwz2"]
Sep 09 16:32:44.093485 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.093341    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca69816-74f7-4bf0-b800-fe10aadfde74" containerName="main"
Sep 09 16:32:44.093485 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.093352    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca69816-74f7-4bf0-b800-fe10aadfde74" containerName="main"
Sep 09 16:32:44.093485 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.093370    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d10e5f-5e4e-411c-b162-7653c4837b0a" containerName="main"
Sep 09 16:32:44.093485 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.093375    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d10e5f-5e4e-411c-b162-7653c4837b0a" containerName="main"
Sep 09 16:32:44.093485 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.093422    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="92d10e5f-5e4e-411c-b162-7653c4837b0a" containerName="main"
Sep 09 16:32:44.093485 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.093432    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ca69816-74f7-4bf0-b800-fe10aadfde74" containerName="main"
Sep 09 16:32:44.096294 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.096277    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g6q5c/must-gather-6xwz2"
Sep 09 16:32:44.098536 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.098515    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g6q5c\"/\"default-dockercfg-92n2g\""
Sep 09 16:32:44.099059 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.099039    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g6q5c\"/\"openshift-service-ca.crt\""
Sep 09 16:32:44.099204 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.099184    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g6q5c\"/\"kube-root-ca.crt\""
Sep 09 16:32:44.103520 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.103497    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g6q5c/must-gather-6xwz2"]
Sep 09 16:32:44.118017 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.117979    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:32:44.217180 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.217147    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log" limit=1024
Sep 09 16:32:44.241636 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.241609    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f7f37237-0c9c-4582-b43a-cae238767f8b-must-gather-output\") pod \"must-gather-6xwz2\" (UID: \"f7f37237-0c9c-4582-b43a-cae238767f8b\") " pod="openshift-must-gather-g6q5c/must-gather-6xwz2"
Sep 09 16:32:44.241749 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.241666    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgkh6\" (UniqueName: \"kubernetes.io/projected/f7f37237-0c9c-4582-b43a-cae238767f8b-kube-api-access-qgkh6\") pod \"must-gather-6xwz2\" (UID: \"f7f37237-0c9c-4582-b43a-cae238767f8b\") " pod="openshift-must-gather-g6q5c/must-gather-6xwz2"
Sep 09 16:32:44.342511 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.342447    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgkh6\" (UniqueName: \"kubernetes.io/projected/f7f37237-0c9c-4582-b43a-cae238767f8b-kube-api-access-qgkh6\") pod \"must-gather-6xwz2\" (UID: \"f7f37237-0c9c-4582-b43a-cae238767f8b\") " pod="openshift-must-gather-g6q5c/must-gather-6xwz2"
Sep 09 16:32:44.342687 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.342563    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f7f37237-0c9c-4582-b43a-cae238767f8b-must-gather-output\") pod \"must-gather-6xwz2\" (UID: \"f7f37237-0c9c-4582-b43a-cae238767f8b\") " pod="openshift-must-gather-g6q5c/must-gather-6xwz2"
Sep 09 16:32:44.342859 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.342838    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f7f37237-0c9c-4582-b43a-cae238767f8b-must-gather-output\") pod \"must-gather-6xwz2\" (UID: \"f7f37237-0c9c-4582-b43a-cae238767f8b\") " pod="openshift-must-gather-g6q5c/must-gather-6xwz2"
Sep 09 16:32:44.350279 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.350220    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgkh6\" (UniqueName: \"kubernetes.io/projected/f7f37237-0c9c-4582-b43a-cae238767f8b-kube-api-access-qgkh6\") pod \"must-gather-6xwz2\" (UID: \"f7f37237-0c9c-4582-b43a-cae238767f8b\") " pod="openshift-must-gather-g6q5c/must-gather-6xwz2"
Sep 09 16:32:44.406321 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.406291    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g6q5c/must-gather-6xwz2"
Sep 09 16:32:44.674039 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.674015    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g6q5c/must-gather-6xwz2"]
Sep 09 16:32:44.676437 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:32:44.676409    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7f37237_0c9c_4582_b43a_cae238767f8b.slice/crio-45c23a24b3d278d3f4e8290abe1daa9979035819f365094f0b73a23f6c45b9ae WatchSource:0}: Error finding container 45c23a24b3d278d3f4e8290abe1daa9979035819f365094f0b73a23f6c45b9ae: Status 404 returned error can't find the container with id 45c23a24b3d278d3f4e8290abe1daa9979035819f365094f0b73a23f6c45b9ae
Sep 09 16:32:44.678353 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:44.678335    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:32:45.305248 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:45.305209    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g6q5c/must-gather-6xwz2" event={"ID":"f7f37237-0c9c-4582-b43a-cae238767f8b","Type":"ContainerStarted","Data":"45c23a24b3d278d3f4e8290abe1daa9979035819f365094f0b73a23f6c45b9ae"}
Sep 09 16:32:46.312301 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:46.312266    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g6q5c/must-gather-6xwz2" event={"ID":"f7f37237-0c9c-4582-b43a-cae238767f8b","Type":"ContainerStarted","Data":"53238b6e03b687adba42a76d883b339880f0115397d63d0609d9a2473d55edcc"}
Sep 09 16:32:46.312746 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:46.312322    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g6q5c/must-gather-6xwz2" event={"ID":"f7f37237-0c9c-4582-b43a-cae238767f8b","Type":"ContainerStarted","Data":"3db0752af7b15d15743d50b1df3dbea93274f070be29468da9fe42c0e53d6682"}
Sep 09 16:32:46.326982 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:46.326922    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g6q5c/must-gather-6xwz2" podStartSLOduration=1.4764374120000001 podStartE2EDuration="2.326904807s" totalImagesPullingTime="850.467395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:32:46.32597966 +0000 UTC m=+2471.192490928" watchObservedRunningTime="2026-09-09 16:32:46.326904807 +0000 UTC m=+2471.193416026"
Sep 09 16:32:47.075108 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:47.075074    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7fdsm_9c5f3c15-9eee-43d1-b403-f89806e864cb/global-pull-secret-syncer/0.log"
Sep 09 16:32:47.215450 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:47.215410    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5jrvg_0880616b-7f54-4880-816e-f21755439fdf/konnectivity-agent/0.log"
Sep 09 16:32:47.327448 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:47.327360    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-93.ec2.internal_a2ab3a812dc27588fde74d62d7cf30f3/haproxy/0.log"
Sep 09 16:32:51.546489 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:51.545751    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6574f9ffbb-tt6rp_56f9152d-41ed-4827-8f66-c3b2b9af5c15/authorino/0.log"
Sep 09 16:32:51.646998 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:51.646934    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9vfgr_bc269cb7-8c85-49e5-96cd-aac33a0f7cf8/kuadrant-console-plugin/0.log"
Sep 09 16:32:53.610147 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:53.610109    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-w2j7v_03edb11a-1a2d-49ea-aacf-3b6c4318a08f/kube-state-metrics/0.log"
Sep 09 16:32:53.629605 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:53.629567    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-w2j7v_03edb11a-1a2d-49ea-aacf-3b6c4318a08f/kube-rbac-proxy-main/0.log"
Sep 09 16:32:53.650868 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:53.650836    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-w2j7v_03edb11a-1a2d-49ea-aacf-3b6c4318a08f/kube-rbac-proxy-self/0.log"
Sep 09 16:32:53.882519 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:53.882423    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zg4bq_8591e18a-a4e5-489c-a3d2-d914123c9bd3/node-exporter/0.log"
Sep 09 16:32:53.910194 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:53.910166    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zg4bq_8591e18a-a4e5-489c-a3d2-d914123c9bd3/kube-rbac-proxy/0.log"
Sep 09 16:32:53.932548 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:53.932516    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zg4bq_8591e18a-a4e5-489c-a3d2-d914123c9bd3/init-textfile/0.log"
Sep 09 16:32:58.983017 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:58.982980    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hdkxg_6833917f-b72a-4586-98ce-c68895a682f1/dns/0.log"
Sep 09 16:32:59.003317 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:59.003289    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hdkxg_6833917f-b72a-4586-98ce-c68895a682f1/kube-rbac-proxy/0.log"
Sep 09 16:32:59.070437 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:59.070400    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dzktt_8d08d290-7a84-4502-8534-3c03383e94f0/dns-node-resolver/0.log"
Sep 09 16:32:59.652791 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:32:59.652756    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kx87f_c1c0dea1-1500-498d-ade5-e8bd61bbcd0b/node-ca/0.log"
Sep 09 16:33:00.839716 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:00.839682    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-747b697884-whtvt_4972d020-567d-464e-8f08-21ca7620cab0/kube-auth-proxy/0.log"
Sep 09 16:33:00.923962 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:00.923927    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-zw24c_d6653c3a-2dbc-41af-8b21-9ca857ba26f4/payload-pre-processing/0.log"
Sep 09 16:33:01.836210 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:01.836182    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9h8tp_48f47946-24d8-4c4a-9915-8b21f4fe4082/serve-healthcheck-canary/0.log"
Sep 09 16:33:02.438427 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:02.438398    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-48skw_3337cbb2-24ee-40cd-8934-2f11d0069e0f/kube-rbac-proxy/0.log"
Sep 09 16:33:02.458314 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:02.458282    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-48skw_3337cbb2-24ee-40cd-8934-2f11d0069e0f/exporter/0.log"
Sep 09 16:33:02.478606 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:02.478563    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-48skw_3337cbb2-24ee-40cd-8934-2f11d0069e0f/extractor/0.log"
Sep 09 16:33:02.976751 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:02.976708    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"]
Sep 09 16:33:02.985832 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:02.985799    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:02.990122 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:02.990094    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"]
Sep 09 16:33:03.023384 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.023315    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e3fb72e-c570-4fc6-8c1d-d07835ddf01c-lib-modules\") pod \"perf-node-gather-daemonset-z2nls\" (UID: \"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.023384 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.023380    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e3fb72e-c570-4fc6-8c1d-d07835ddf01c-podres\") pod \"perf-node-gather-daemonset-z2nls\" (UID: \"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.023656 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.023423    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e3fb72e-c570-4fc6-8c1d-d07835ddf01c-proc\") pod \"perf-node-gather-daemonset-z2nls\" (UID: \"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.023656 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.023504    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d8bf\" (UniqueName: \"kubernetes.io/projected/3e3fb72e-c570-4fc6-8c1d-d07835ddf01c-kube-api-access-4d8bf\") pod \"perf-node-gather-daemonset-z2nls\" (UID: \"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.023656 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.023549    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e3fb72e-c570-4fc6-8c1d-d07835ddf01c-sys\") pod \"perf-node-gather-daemonset-z2nls\" (UID: \"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.124553 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.124519    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8bf\" (UniqueName: \"kubernetes.io/projected/3e3fb72e-c570-4fc6-8c1d-d07835ddf01c-kube-api-access-4d8bf\") pod \"perf-node-gather-daemonset-z2nls\" (UID: \"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.124727 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.124564    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e3fb72e-c570-4fc6-8c1d-d07835ddf01c-sys\") pod \"perf-node-gather-daemonset-z2nls\" (UID: \"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.124727 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.124626    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e3fb72e-c570-4fc6-8c1d-d07835ddf01c-lib-modules\") pod \"perf-node-gather-daemonset-z2nls\" (UID: \"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.124727 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.124659    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e3fb72e-c570-4fc6-8c1d-d07835ddf01c-podres\") pod \"perf-node-gather-daemonset-z2nls\" (UID: \"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.124727 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.124711    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e3fb72e-c570-4fc6-8c1d-d07835ddf01c-sys\") pod \"perf-node-gather-daemonset-z2nls\" (UID: \"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.124932 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.124753    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e3fb72e-c570-4fc6-8c1d-d07835ddf01c-proc\") pod \"perf-node-gather-daemonset-z2nls\" (UID: \"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.124932 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.124789    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e3fb72e-c570-4fc6-8c1d-d07835ddf01c-podres\") pod \"perf-node-gather-daemonset-z2nls\" (UID: \"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.124932 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.124809    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e3fb72e-c570-4fc6-8c1d-d07835ddf01c-lib-modules\") pod \"perf-node-gather-daemonset-z2nls\" (UID: \"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.124932 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.124844    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e3fb72e-c570-4fc6-8c1d-d07835ddf01c-proc\") pod \"perf-node-gather-daemonset-z2nls\" (UID: \"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.133536 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.133504    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d8bf\" (UniqueName: \"kubernetes.io/projected/3e3fb72e-c570-4fc6-8c1d-d07835ddf01c-kube-api-access-4d8bf\") pod \"perf-node-gather-daemonset-z2nls\" (UID: \"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.299604 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.299508    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:03.583595 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:03.583561    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"]
Sep 09 16:33:03.587355 ip-10-0-133-93 kubenswrapper[2556]: W0909 16:33:03.587323    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3e3fb72e_c570_4fc6_8c1d_d07835ddf01c.slice/crio-f4881d12084dc192f68cf29da4be518ebbe86c8d14f8a6ed33f4f1df16b0973b WatchSource:0}: Error finding container f4881d12084dc192f68cf29da4be518ebbe86c8d14f8a6ed33f4f1df16b0973b: Status 404 returned error can't find the container with id f4881d12084dc192f68cf29da4be518ebbe86c8d14f8a6ed33f4f1df16b0973b
Sep 09 16:33:04.412656 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:04.412611    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls" event={"ID":"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c","Type":"ContainerStarted","Data":"1147aff565f6437f871c574467d23e9e4b56cb490d830c5ce814a89af8285981"}
Sep 09 16:33:04.412852 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:04.412724    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls" event={"ID":"3e3fb72e-c570-4fc6-8c1d-d07835ddf01c","Type":"ContainerStarted","Data":"f4881d12084dc192f68cf29da4be518ebbe86c8d14f8a6ed33f4f1df16b0973b"}
Sep 09 16:33:04.412852 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:04.412755    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:04.426947 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:04.426883    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls" podStartSLOduration=2.426863705 podStartE2EDuration="2.426863705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:33:04.426816985 +0000 UTC m=+2489.293328205" watchObservedRunningTime="2026-09-09 16:33:04.426863705 +0000 UTC m=+2489.293374926"
Sep 09 16:33:05.253961 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:05.253922    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4ccd886c-lfmxt_efca11da-bf8c-469c-b9e1-904559b5c395/manager/0.log"
Sep 09 16:33:05.274117 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:05.274079    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4ccd886c-lfmxt_efca11da-bf8c-469c-b9e1-904559b5c395/copy-manifests/0.log"
Sep 09 16:33:06.470290 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:06.470253    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-678f94cf5f-gw5mr_d0de56ca-b8ec-4087-ae6f-16e839dfcebb/manager/0.log"
Sep 09 16:33:06.522791 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:06.522750    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-sglz8_1fa70f62-3193-4fdb-872d-a7ec355d18b9/openshift-lws-operator/0.log"
Sep 09 16:33:07.679998 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:07.679966    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-747b697884-whtvt_4972d020-567d-464e-8f08-21ca7620cab0/kube-auth-proxy/0.log"
Sep 09 16:33:07.744321 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:07.744283    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-zw24c_d6653c3a-2dbc-41af-8b21-9ca857ba26f4/payload-pre-processing/0.log"
Sep 09 16:33:09.459273 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:09.459236    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-747b697884-whtvt_4972d020-567d-464e-8f08-21ca7620cab0/kube-auth-proxy/0.log"
Sep 09 16:33:09.530433 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:09.530395    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-zw24c_d6653c3a-2dbc-41af-8b21-9ca857ba26f4/payload-pre-processing/0.log"
Sep 09 16:33:10.432843 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:10.432808    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-z2nls"
Sep 09 16:33:13.435850 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:13.435825    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xp9wv_322726ce-594b-4b74-968b-40ad101cb1fb/kube-multus-additional-cni-plugins/0.log"
Sep 09 16:33:13.455023 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:13.455000    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xp9wv_322726ce-594b-4b74-968b-40ad101cb1fb/egress-router-binary-copy/0.log"
Sep 09 16:33:13.474356 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:13.474333    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xp9wv_322726ce-594b-4b74-968b-40ad101cb1fb/cni-plugins/0.log"
Sep 09 16:33:13.493854 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:13.493825    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xp9wv_322726ce-594b-4b74-968b-40ad101cb1fb/bond-cni-plugin/0.log"
Sep 09 16:33:13.513607 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:13.513585    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xp9wv_322726ce-594b-4b74-968b-40ad101cb1fb/routeoverride-cni/0.log"
Sep 09 16:33:13.533164 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:13.533140    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xp9wv_322726ce-594b-4b74-968b-40ad101cb1fb/whereabouts-cni-bincopy/0.log"
Sep 09 16:33:13.552164 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:13.552142    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xp9wv_322726ce-594b-4b74-968b-40ad101cb1fb/whereabouts-cni/0.log"
Sep 09 16:33:13.616675 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:13.616636    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lxvn6_cc5d1009-a15f-4c28-b291-074695366f79/kube-multus/0.log"
Sep 09 16:33:13.678221 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:13.678191    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4tmj4_72246d50-b16d-4f50-b064-aa2a85cf2970/network-metrics-daemon/0.log"
Sep 09 16:33:13.697114 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:13.697022    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4tmj4_72246d50-b16d-4f50-b064-aa2a85cf2970/kube-rbac-proxy/0.log"
Sep 09 16:33:14.899370 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:14.899278    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-controller/0.log"
Sep 09 16:33:14.914344 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:14.914316    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/0.log"
Sep 09 16:33:14.914498 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:14.914383    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovn-acl-logging/1.log"
Sep 09 16:33:14.938569 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:14.938544    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/kube-rbac-proxy-node/0.log"
Sep 09 16:33:14.963453 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:14.963433    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 16:33:14.980642 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:14.980617    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/northd/0.log"
Sep 09 16:33:14.999601 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:14.999582    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/nbdb/0.log"
Sep 09 16:33:15.019340 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:15.019318    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/sbdb/0.log"
Sep 09 16:33:15.109514 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:15.109484    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms57r_80563802-ee38-4620-addd-926a31290be2/ovnkube-controller/0.log"
Sep 09 16:33:16.460097 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:16.460068    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-xsht8_7bafb10d-a48d-4429-b1a4-532d3ed5a475/network-check-target-container/0.log"
Sep 09 16:33:17.589704 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:17.589679    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-f47qf_d8627dab-9db2-4cb5-9ee9-37c644998bbb/iptables-alerter/0.log"
Sep 09 16:33:18.299353 ip-10-0-133-93 kubenswrapper[2556]: I0909 16:33:18.299323    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-wv8m7_63f36a9e-8e7a-49cf-b8f6-635aa633bfa9/tuned/0.log"